[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:15 dist-test-slave-9x3b krb5kdc[14810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:15 dist-test-slave-9x3b krb5kdc[14810](info): set up 2 sockets
Jun 19 06:48:15 dist-test-slave-9x3b krb5kdc[14810](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 06:48:16.463526 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:16 dist-test-slave-9x3b krb5kdc[14810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:16.576695 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:43045
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:43045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:16.683298 14823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:16.683569 14823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:16.683667 14823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:16.687726 14823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:16.687806 14823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:16.687850 14823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:16.687880 14823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:16.687911 14823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:16.693099 14823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:43045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:43045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:16.694103 14823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:16.695111 14823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:16.703641 14831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:16.703672 14828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:16.703687 14823 server_base.cc:1034] running on GCE node
W20260619 06:48:16.703670 14829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:16.704443 14823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:16.705520 14823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:16.706703 14823 hybrid_clock.cc:648] HybridClock initialized: now 1781851696706688 us; error 28 us; skew 500 ppm
Jun 19 06:48:16 dist-test-slave-9x3b krb5kdc[14810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851696, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:16.709412 14823 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:16.710276 14823 webserver.cc:460] Webserver started at http://127.14.116.254:35947/ using document root <none> and password file <none>
I20260619 06:48:16.710834 14823 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:16.710922 14823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:16.711191 14823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:16.713336 14823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "dfededdcdfb34e648af8a8b747309857"
format_stamp: "Formatted at 2026-06-19 06:48:16 on dist-test-slave-9x3b"
server_key: "e2b40215fcd886aaad2dd012cee455bc"
server_key_iv: "bfca85a36f6c0e0231082f8499744bde"
server_key_version: "encryptionkey@0"
I20260619 06:48:16.713891 14823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "dfededdcdfb34e648af8a8b747309857"
format_stamp: "Formatted at 2026-06-19 06:48:16 on dist-test-slave-9x3b"
server_key: "e2b40215fcd886aaad2dd012cee455bc"
server_key_iv: "bfca85a36f6c0e0231082f8499744bde"
server_key_version: "encryptionkey@0"
I20260619 06:48:16.717559 14823 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 06:48:16.719998 14838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:16.721184 14823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:16.721331 14823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "dfededdcdfb34e648af8a8b747309857"
format_stamp: "Formatted at 2026-06-19 06:48:16 on dist-test-slave-9x3b"
server_key: "e2b40215fcd886aaad2dd012cee455bc"
server_key_iv: "bfca85a36f6c0e0231082f8499744bde"
server_key_version: "encryptionkey@0"
I20260619 06:48:16.721437 14823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:16.731918 14823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:16.734987 14823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:16.735172 14823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:16.742760 14823 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:43045
I20260619 06:48:16.742796 14890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:43045 every 8 connection(s)
I20260619 06:48:16.743796 14823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:16.746447 14891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:16.751806 14891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857: Bootstrap starting.
I20260619 06:48:16.752312 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 14823
I20260619 06:48:16.752454 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:16.753659 14803 external_mini_cluster.cc:1428] Setting key c89e283fd6f2ac808707fa38e4ce7f96
I20260619 06:48:16.754417 14891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:16.755148 14891 log.cc:824] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:16.757414 14891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857: No bootstrap required, opened a new log
I20260619 06:48:16.761369 14891 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfededdcdfb34e648af8a8b747309857" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43045 } }
I20260619 06:48:16.761600 14891 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:16.761698 14891 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfededdcdfb34e648af8a8b747309857, State: Initialized, Role: FOLLOWER
I20260619 06:48:16.762202 14891 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfededdcdfb34e648af8a8b747309857" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43045 } }
I20260619 06:48:16.762363 14891 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:16.762454 14891 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:16.762554 14891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:16.763404 14891 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfededdcdfb34e648af8a8b747309857" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43045 } }
I20260619 06:48:16.763792 14891 leader_election.cc:302] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfededdcdfb34e648af8a8b747309857; no voters:
I20260619 06:48:16.764101 14891 leader_election.cc:288] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:16.764200 14896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:16.764441 14896 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [term 1 LEADER]: Becoming Leader. State: Replica: dfededdcdfb34e648af8a8b747309857, State: Running, Role: LEADER
I20260619 06:48:16.764765 14896 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfededdcdfb34e648af8a8b747309857" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43045 } }
I20260619 06:48:16.765286 14891 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:16.766404 14897 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfededdcdfb34e648af8a8b747309857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfededdcdfb34e648af8a8b747309857" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43045 } } }
I20260619 06:48:16.766520 14897 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:16.766891 14898 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfededdcdfb34e648af8a8b747309857. Latest consensus state: current_term: 1 leader_uuid: "dfededdcdfb34e648af8a8b747309857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfededdcdfb34e648af8a8b747309857" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43045 } } }
I20260619 06:48:16.767056 14898 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857 [sys.catalog]: This master's current role is: LEADER
W20260619 06:48:16.770951 14912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:48:16.771025 14912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:48:16.771096 14905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:16.772054 14905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 06:48:16 dist-test-slave-9x3b krb5kdc[14810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:16.777664 14905 catalog_manager.cc:1347] Generated new cluster ID: 93872c1ad6c7481db7a98f24917e2854
I20260619 06:48:16.777724 14905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:16.778853 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:16.756537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.14.116.254:43045)
0619 06:48:16.756920 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:16.756936 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:16.758163 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 06:48:16.760995 (+ 2832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:16.761010 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:16.761316 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:16.761629 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:16.764029 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:16.765427 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:16.767127 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:16.767850 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:16.768000 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:16.773094 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:16.773101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:16.773118 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:16.773148 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:16.775308 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:16.775980 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:16.775986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:16.775990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:16.776099 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:16.776504 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:16.776509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:16.776512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:16.777070 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:16.777297 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:16.777931 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 06:48:16.778185 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":120,"threads_started":1}
I20260619 06:48:16.799515 14905 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:16.800580 14905 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:16.809552 14905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfededdcdfb34e648af8a8b747309857: Generated new TSK 0
I20260619 06:48:16.810237 14905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:16.838760 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43045
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:16.953163 14917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:16.953397 14917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:16.953438 14917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:16.957440 14917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:16.957495 14917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:16.957551 14917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:16.962706 14917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43045
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:16.963581 14917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:16.964607 14917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:16.973448 14925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:16.973630 14917 server_base.cc:1034] running on GCE node
W20260619 06:48:16.973446 14922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:16.973480 14923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:16.974116 14917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:16.974715 14917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:16.975870 14917 hybrid_clock.cc:648] HybridClock initialized: now 1781851696975818 us; error 63 us; skew 500 ppm
Jun 19 06:48:16 dist-test-slave-9x3b krb5kdc[14810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851696, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:16.978487 14917 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:16.979295 14917 webserver.cc:460] Webserver started at http://127.14.116.193:41967/ using document root <none> and password file <none>
I20260619 06:48:16.979883 14917 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:16.979943 14917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:16.980155 14917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:16.982010 14917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "049897e866f543ceb6bd7962bcfb4de4"
format_stamp: "Formatted at 2026-06-19 06:48:16 on dist-test-slave-9x3b"
server_key: "6165338241803491295a97752c171f7a"
server_key_iv: "641e65d2654b399f66743f779f3c92a3"
server_key_version: "encryptionkey@0"
I20260619 06:48:16.982555 14917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "049897e866f543ceb6bd7962bcfb4de4"
format_stamp: "Formatted at 2026-06-19 06:48:16 on dist-test-slave-9x3b"
server_key: "6165338241803491295a97752c171f7a"
server_key_iv: "641e65d2654b399f66743f779f3c92a3"
server_key_version: "encryptionkey@0"
I20260619 06:48:16.986009 14917 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 06:48:16.988217 14932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:16.989346 14917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:16.989439 14917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "049897e866f543ceb6bd7962bcfb4de4"
format_stamp: "Formatted at 2026-06-19 06:48:16 on dist-test-slave-9x3b"
server_key: "6165338241803491295a97752c171f7a"
server_key_iv: "641e65d2654b399f66743f779f3c92a3"
server_key_version: "encryptionkey@0"
I20260619 06:48:16.989518 14917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:16.999678 14917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:17.002215 14917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:17.002355 14917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:17.002885 14917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:17.003683 14917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:17.003739 14917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:17.003777 14917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:17.003798 14917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:17.012006 14917 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:40879
I20260619 06:48:17.012032 15045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:40879 every 8 connection(s)
I20260619 06:48:17.012854 14917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:17.014540 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 14917
I20260619 06:48:17.014642 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:17.014911 14803 external_mini_cluster.cc:1428] Setting key 4b4f19a86baa1ebb0370bd5f063d3550
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[14810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851696, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
I20260619 06:48:17.026682 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:17.014623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:35781 (local address 127.14.116.254:43045)
0619 06:48:17.015443 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:17.015448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:17.016056 (+ 608us) server_negotiation.cc:407] Connection header received
0619 06:48:17.016989 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:17.016995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:17.017058 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:17.017149 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:17.018275 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.018797 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.019980 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.020208 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.020315 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:17.023051 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:17.023054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:17.023056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:17.023078 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:17.024674 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.025138 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.025141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.025142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.025261 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.025556 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.025558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.025559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.025756 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:17.025876 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:17.026384 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 06:48:17.026514 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725}
I20260619 06:48:17.027606 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:17.015060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:43045 (local address 127.14.116.193:35781)
0619 06:48:17.015901 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:17.015955 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:17.016784 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:17.017289 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:17.017296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:17.017662 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:17.018133 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:17.018146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.018923 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.018926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:17.019857 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:17.019867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.020346 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.020350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:17.020523 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:17.021091 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:17.021106 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:17.022883 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:17.024808 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:17.024813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:17.024824 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:17.025056 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:17.025338 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:17.025340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:17.025342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:17.025478 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:17.025916 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:17.025922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:17.026142 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 06:48:17.026321 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 06:48:17.026625 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":145,"threads_started":1}
I20260619 06:48:17.028847 15046 heartbeater.cc:347] Connected to a master server at 127.14.116.254:43045
I20260619 06:48:17.029093 15046 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:17.029685 15046 heartbeater.cc:511] Master 127.14.116.254:43045 requested a full tablet report, sending...
I20260619 06:48:17.031216 14855 ts_manager.cc:194] Registered new tserver with Master: 049897e866f543ceb6bd7962bcfb4de4 (127.14.116.193:40879)
I20260619 06:48:17.032397 14855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:35781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:17.042068 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43045
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:17.153347 15051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:17.153601 15051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:17.153643 15051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:17.157646 15051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:17.157703 15051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:17.157783 15051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:17.163043 15051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43045
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:17.164095 15051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:17.165141 15051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:17.174427 15057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:17.174593 15051 server_base.cc:1034] running on GCE node
W20260619 06:48:17.174446 15059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:17.174446 15056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:17.175120 15051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:17.175778 15051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:17.176978 15051 hybrid_clock.cc:648] HybridClock initialized: now 1781851697176940 us; error 45 us; skew 500 ppm
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[14810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851697, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:17.179528 15051 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:17.180330 15051 webserver.cc:460] Webserver started at http://127.14.116.194:46859/ using document root <none> and password file <none>
I20260619 06:48:17.180811 15051 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:17.180866 15051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:17.181020 15051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:17.182893 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "d1a7dcd68fb8476b8748287b9f424d05"
format_stamp: "Formatted at 2026-06-19 06:48:17 on dist-test-slave-9x3b"
server_key: "c5cf0c1cba3af0bf7915160a14687f25"
server_key_iv: "297e9adc79514c00ebb6797df8aafbf5"
server_key_version: "encryptionkey@0"
I20260619 06:48:17.183398 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "d1a7dcd68fb8476b8748287b9f424d05"
format_stamp: "Formatted at 2026-06-19 06:48:17 on dist-test-slave-9x3b"
server_key: "c5cf0c1cba3af0bf7915160a14687f25"
server_key_iv: "297e9adc79514c00ebb6797df8aafbf5"
server_key_version: "encryptionkey@0"
I20260619 06:48:17.186837 15051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:48:17.189079 15066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:17.190274 15051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:17.190371 15051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "d1a7dcd68fb8476b8748287b9f424d05"
format_stamp: "Formatted at 2026-06-19 06:48:17 on dist-test-slave-9x3b"
server_key: "c5cf0c1cba3af0bf7915160a14687f25"
server_key_iv: "297e9adc79514c00ebb6797df8aafbf5"
server_key_version: "encryptionkey@0"
I20260619 06:48:17.190451 15051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:17.213678 15051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:17.216393 15051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:17.216537 15051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:17.217033 15051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:17.217765 15051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:17.217814 15051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:17.217852 15051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:17.217867 15051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:17.226514 15051 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:39473
I20260619 06:48:17.226545 15179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:39473 every 8 connection(s)
I20260619 06:48:17.227337 15051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:17.230036 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 15051
I20260619 06:48:17.230157 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:17.230448 14803 external_mini_cluster.cc:1428] Setting key efe526369010da95533f3c203e42550f
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[14810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851697, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:48:17.240743 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:17.228976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:42091 (local address 127.14.116.254:43045)
0619 06:48:17.229108 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:17.229113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:17.230210 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 06:48:17.231289 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:17.231292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:17.231338 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:17.231421 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:17.232907 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.233427 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.234571 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.234791 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.234910 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:17.237282 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:17.237285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:17.237287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:17.237315 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:17.238809 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.239278 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.239283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.239285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.239365 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.239721 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.239725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.239726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.239898 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:17.240005 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:17.240461 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 06:48:17.240578 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:48:17.241466 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:17.229357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:43045 (local address 127.14.116.194:42091)
0619 06:48:17.230049 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:17.230107 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:17.231073 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:17.231564 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:17.231570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:17.232091 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:17.232748 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:17.232768 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.233565 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.233568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:17.234435 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:17.234444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.234938 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.234942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:17.235107 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:17.235716 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:17.235731 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:17.237120 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:17.238916 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:17.238923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:17.238936 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:17.239178 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:17.239461 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:17.239465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:17.239467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:17.239597 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:17.240001 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:17.240007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:17.240236 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 06:48:17.240410 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 06:48:17.240700 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:48:17.242537 15180 heartbeater.cc:347] Connected to a master server at 127.14.116.254:43045
I20260619 06:48:17.242774 15180 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:17.243317 15180 heartbeater.cc:511] Master 127.14.116.254:43045 requested a full tablet report, sending...
I20260619 06:48:17.244375 14855 ts_manager.cc:194] Registered new tserver with Master: d1a7dcd68fb8476b8748287b9f424d05 (127.14.116.194:39473)
I20260619 06:48:17.244927 14855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:42091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:17.256690 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43045
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:17.362402 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:17.362668 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:17.362741 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:17.366981 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:17.367051 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:17.367138 15185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:17.372401 15185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43045
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:17.373344 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:17.374307 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:17.384063 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:17.384119 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:17.384104 15185 server_base.cc:1034] running on GCE node
W20260619 06:48:17.384213 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:17.384761 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:17.385366 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:17.386523 15185 hybrid_clock.cc:648] HybridClock initialized: now 1781851697386509 us; error 25 us; skew 500 ppm
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[14810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851697, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:17.389235 15185 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:17.390048 15185 webserver.cc:460] Webserver started at http://127.14.116.195:33201/ using document root <none> and password file <none>
I20260619 06:48:17.390586 15185 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:17.390645 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:17.390884 15185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:17.392982 15185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "b27f3abbabee4c58a07a3b4329b9a3be"
format_stamp: "Formatted at 2026-06-19 06:48:17 on dist-test-slave-9x3b"
server_key: "2324eae026cd16a5d17fa8d823795a87"
server_key_iv: "46d378d52ee7637a9c904b9ad292b91b"
server_key_version: "encryptionkey@0"
I20260619 06:48:17.393536 15185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "b27f3abbabee4c58a07a3b4329b9a3be"
format_stamp: "Formatted at 2026-06-19 06:48:17 on dist-test-slave-9x3b"
server_key: "2324eae026cd16a5d17fa8d823795a87"
server_key_iv: "46d378d52ee7637a9c904b9ad292b91b"
server_key_version: "encryptionkey@0"
I20260619 06:48:17.397217 15185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 06:48:17.399430 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:17.400650 15185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:17.400776 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "b27f3abbabee4c58a07a3b4329b9a3be"
format_stamp: "Formatted at 2026-06-19 06:48:17 on dist-test-slave-9x3b"
server_key: "2324eae026cd16a5d17fa8d823795a87"
server_key_iv: "46d378d52ee7637a9c904b9ad292b91b"
server_key_version: "encryptionkey@0"
I20260619 06:48:17.400892 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:17.415143 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:17.418354 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:17.418548 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:17.419116 15185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:17.419947 15185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:17.420014 15185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:17.420086 15185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:17.420128 15185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:17.428929 15185 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:33859
I20260619 06:48:17.428957 15313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:33859 every 8 connection(s)
I20260619 06:48:17.429747 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:17.431815 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 15185
I20260619 06:48:17.431896 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:17.432248 14803 external_mini_cluster.cc:1428] Setting key 090ec0ca0ce73c8ffb5582f2095370ad
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[14810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851697, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:17.442195 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:17.431414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:43167 (local address 127.14.116.254:43045)
0619 06:48:17.431555 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:17.431559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:17.432753 (+ 1194us) server_negotiation.cc:407] Connection header received
0619 06:48:17.433889 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:17.433892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:17.433943 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:17.434024 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:17.435290 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.435892 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.436942 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.437145 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.437219 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:17.439378 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:17.439381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:17.439383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:17.439404 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:17.440627 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.441046 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.441049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.441050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.441114 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.441381 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.441383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.441384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.441530 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:17.441601 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:17.442002 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 06:48:17.442086 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:48:17.442943 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:17.431781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:43045 (local address 127.14.116.195:43167)
0619 06:48:17.432567 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:17.432621 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:17.433624 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:17.434177 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:17.434183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:17.434620 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:17.435163 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:17.435176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.435990 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.435993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:17.436840 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:17.436848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.437263 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.437266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:17.437394 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:17.437888 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:17.437902 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:17.439236 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:17.440720 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:17.440725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:17.440741 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:17.440967 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:17.441187 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:17.441189 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:17.441191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:17.441308 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:17.441611 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:17.441615 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:17.441816 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 06:48:17.441960 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 06:48:17.442170 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":135,"threads_started":1}
I20260619 06:48:17.444018 15314 heartbeater.cc:347] Connected to a master server at 127.14.116.254:43045
I20260619 06:48:17.444227 15314 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:17.444657 15314 heartbeater.cc:511] Master 127.14.116.254:43045 requested a full tablet report, sending...
I20260619 06:48:17.445662 14855 ts_manager.cc:194] Registered new tserver with Master: b27f3abbabee4c58a07a3b4329b9a3be (127.14.116.195:33859)
I20260619 06:48:17.446190 14855 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:43167
I20260619 06:48:17.446274 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[14810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[14810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:17.471118 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:17.461935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.14.116.193:40879)
0619 06:48:17.462237 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:17.462255 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:17.462277 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:48:17.462324 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:17.462328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:17.462491 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:17.462603 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:17.463215 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.464049 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.464893 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.465239 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.465363 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:17.467944 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:17.467952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:17.467963 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:17.467985 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:17.469374 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.469807 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.469815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.469817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.469907 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.470176 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.470181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.470183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.470418 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:17.470624 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:17.470794 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 06:48:17.470941 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":133,"threads_started":1}
W20260619 06:48:17.472110 14960 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34848
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[14810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[14810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:17.495694 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:17.488140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.14.116.193:40879)
0619 06:48:17.488263 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:17.488267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:17.488438 (+ 171us) server_negotiation.cc:407] Connection header received
0619 06:48:17.488538 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:17.488540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:17.488578 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:17.488648 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:17.489251 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.489736 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.490472 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:17.490684 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:17.490755 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:17.492982 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:17.492985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:17.492987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:17.493008 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:17.494394 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.494741 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.494743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.494744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.494819 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:17.495028 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:17.495030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:17.495031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:17.495169 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:17.495259 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:17.495444 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 06:48:17.495532 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:48:17.496562 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 14917
I20260619 06:48:17.502866 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 15051
I20260619 06:48:17.509644 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 15185
I20260619 06:48:17.514976 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 14823
2026-06-19T06:48:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2100 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[15328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[15328](info): set up 2 sockets
Jun 19 06:48:17 dist-test-slave-9x3b krb5kdc[15328](info): commencing operation
krb5kdc: starting...
W20260619 06:48:18.547187 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:18 dist-test-slave-9x3b krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:18.658320 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:42191
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:42191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:18.762005 15341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:18.762272 15341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:18.762346 15341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:18.766397 15341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:18.766476 15341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:18.766511 15341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:18.766566 15341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:18.766601 15341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:18.771598 15341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:42191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:42191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:18.772550 15341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:18.773489 15341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:18.781241 15347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:18.781276 15349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:18.781215 15346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:18.781381 15341 server_base.cc:1034] running on GCE node
I20260619 06:48:18.782006 15341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:18.783097 15341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:18.784274 15341 hybrid_clock.cc:648] HybridClock initialized: now 1781851698784261 us; error 24 us; skew 500 ppm
Jun 19 06:48:18 dist-test-slave-9x3b krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851698, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:18.786777 15341 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:18.787551 15341 webserver.cc:460] Webserver started at http://127.14.116.254:38321/ using document root <none> and password file <none>
I20260619 06:48:18.788091 15341 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:18.788149 15341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:18.788297 15341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:18.790421 15341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "9683c0b87bc14db59872d242b23bf86b"
format_stamp: "Formatted at 2026-06-19 06:48:18 on dist-test-slave-9x3b"
server_key: "ff059e0d6c7fe5a8fdd62e38c716a2ea"
server_key_iv: "fec110aab657c9c5d487e8cc807cb127"
server_key_version: "encryptionkey@0"
I20260619 06:48:18.790899 15341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "9683c0b87bc14db59872d242b23bf86b"
format_stamp: "Formatted at 2026-06-19 06:48:18 on dist-test-slave-9x3b"
server_key: "ff059e0d6c7fe5a8fdd62e38c716a2ea"
server_key_iv: "fec110aab657c9c5d487e8cc807cb127"
server_key_version: "encryptionkey@0"
I20260619 06:48:18.794400 15341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 06:48:18.796589 15356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:18.797714 15341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:18.797806 15341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "9683c0b87bc14db59872d242b23bf86b"
format_stamp: "Formatted at 2026-06-19 06:48:18 on dist-test-slave-9x3b"
server_key: "ff059e0d6c7fe5a8fdd62e38c716a2ea"
server_key_iv: "fec110aab657c9c5d487e8cc807cb127"
server_key_version: "encryptionkey@0"
I20260619 06:48:18.797885 15341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:18.812695 15341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:18.815167 15341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:18.815299 15341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:18.821863 15341 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:42191
I20260619 06:48:18.821892 15408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:42191 every 8 connection(s)
I20260619 06:48:18.822731 15341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:18.823570 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 15341
I20260619 06:48:18.823710 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:18.823982 14803 external_mini_cluster.cc:1428] Setting key d52fb4274655cf82d7fc0412ed3c88c0
I20260619 06:48:18.825646 15409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:18.831853 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b: Bootstrap starting.
Jun 19 06:48:18 dist-test-slave-9x3b krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:18.834125 15409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:18.834801 15409 log.cc:824] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:18.836525 15409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b: No bootstrap required, opened a new log
I20260619 06:48:18.837450 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:18.825201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.14.116.254:42191)
0619 06:48:18.825593 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:18.825617 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:18.825649 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:48:18.826284 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:18.826299 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:18.826602 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:18.826857 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:18.827707 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:18.828656 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:18.829760 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:18.830211 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:18.830325 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:18.833030 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:18.833034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:18.833046 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:18.833070 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:18.834704 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:18.835150 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:18.835160 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:18.835164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:18.835264 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:18.835550 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:18.835555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:18.835557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:18.836021 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:18.836216 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:18.836468 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 06:48:18.836773 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":132,"threads_started":1}
I20260619 06:48:18.839329 15409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9683c0b87bc14db59872d242b23bf86b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42191 } }
I20260619 06:48:18.839546 15409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:18.839644 15409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9683c0b87bc14db59872d242b23bf86b, State: Initialized, Role: FOLLOWER
I20260619 06:48:18.840086 15409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9683c0b87bc14db59872d242b23bf86b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42191 } }
I20260619 06:48:18.840231 15409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:18.840313 15409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:18.840405 15409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:18.841303 15409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9683c0b87bc14db59872d242b23bf86b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42191 } }
I20260619 06:48:18.841647 15409 leader_election.cc:302] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9683c0b87bc14db59872d242b23bf86b; no voters:
I20260619 06:48:18.841943 15409 leader_election.cc:288] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:18.842056 15414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:18.842283 15414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [term 1 LEADER]: Becoming Leader. State: Replica: 9683c0b87bc14db59872d242b23bf86b, State: Running, Role: LEADER
I20260619 06:48:18.842602 15414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9683c0b87bc14db59872d242b23bf86b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42191 } }
I20260619 06:48:18.843020 15409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:18.844177 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9683c0b87bc14db59872d242b23bf86b. Latest consensus state: current_term: 1 leader_uuid: "9683c0b87bc14db59872d242b23bf86b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9683c0b87bc14db59872d242b23bf86b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42191 } } }
I20260619 06:48:18.844197 15415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9683c0b87bc14db59872d242b23bf86b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9683c0b87bc14db59872d242b23bf86b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42191 } } }
I20260619 06:48:18.844426 15415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:18.844453 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:18.844798 15424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:18.847381 15424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:18.852480 15424 catalog_manager.cc:1347] Generated new cluster ID: 5a4d564c88cd42a9acef4bf0b97aa4ca
I20260619 06:48:18.852548 15424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:18.874758 15424 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:18.875849 15424 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:18.881839 15424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9683c0b87bc14db59872d242b23bf86b: Generated new TSK 0
I20260619 06:48:18.882489 15424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:18.910274 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:42191
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:19.016151 15435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:19.016383 15435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:19.016424 15435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:19.020232 15435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:19.020274 15435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:19.020328 15435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:19.025277 15435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:42191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:19.026127 15435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:19.027005 15435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:19.035848 15440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:19.036056 15435 server_base.cc:1034] running on GCE node
W20260619 06:48:19.035888 15443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:19.035866 15441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:19.036525 15435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:19.037189 15435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:19.038401 15435 hybrid_clock.cc:648] HybridClock initialized: now 1781851699038383 us; error 25 us; skew 500 ppm
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:19.041090 15435 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:19.041873 15435 webserver.cc:460] Webserver started at http://127.14.116.193:39393/ using document root <none> and password file <none>
I20260619 06:48:19.042397 15435 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:19.042483 15435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:19.042701 15435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:19.044777 15435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "e02e2455210c4189b7ab53d6c2a3cdbf"
format_stamp: "Formatted at 2026-06-19 06:48:19 on dist-test-slave-9x3b"
server_key: "71fb957b13d53b367794b419b34dd9af"
server_key_iv: "3c0a2e1e28b216d734ffcb0d34ac869e"
server_key_version: "encryptionkey@0"
I20260619 06:48:19.045334 15435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "e02e2455210c4189b7ab53d6c2a3cdbf"
format_stamp: "Formatted at 2026-06-19 06:48:19 on dist-test-slave-9x3b"
server_key: "71fb957b13d53b367794b419b34dd9af"
server_key_iv: "3c0a2e1e28b216d734ffcb0d34ac869e"
server_key_version: "encryptionkey@0"
I20260619 06:48:19.048749 15435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:19.050963 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:19.052094 15435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:19.052220 15435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "e02e2455210c4189b7ab53d6c2a3cdbf"
format_stamp: "Formatted at 2026-06-19 06:48:19 on dist-test-slave-9x3b"
server_key: "71fb957b13d53b367794b419b34dd9af"
server_key_iv: "3c0a2e1e28b216d734ffcb0d34ac869e"
server_key_version: "encryptionkey@0"
I20260619 06:48:19.052320 15435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:19.071225 15435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:19.073889 15435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:19.074075 15435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:19.074628 15435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:19.075387 15435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:19.075453 15435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:19.075520 15435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:19.075573 15435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:19.084501 15435 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:45157
I20260619 06:48:19.085425 15435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:19.085799 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 15435
I20260619 06:48:19.085919 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:19.086221 14803 external_mini_cluster.cc:1428] Setting key 5bd1bf5139ff111c5dbe9e339967f385
I20260619 06:48:19.086261 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:45157 every 8 connection(s)
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
I20260619 06:48:19.098898 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.087230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:59739 (local address 127.14.116.254:42191)
0619 06:48:19.087345 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.087349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.088375 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 06:48:19.089637 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.089641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.089694 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.089765 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:19.090852 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.091380 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.092552 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.092783 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.092893 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.095213 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:19.095217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:19.095219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:19.095240 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:19.096818 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.097379 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.097382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.097383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.097482 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.097802 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.097808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.097813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.098062 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:19.098169 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.098608 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.098728 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:48:19.099741 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.087660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:42191 (local address 127.14.116.193:59739)
0619 06:48:19.088206 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:19.088263 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:19.089433 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:19.089940 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:19.089947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:19.090269 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:19.090714 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.090727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.091500 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.091504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.092430 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.092439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.092934 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.092938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.093108 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.093683 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:19.093698 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:19.095053 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:19.096922 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:19.096927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:19.096939 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:19.097283 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:19.097579 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:19.097582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:19.097583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:19.097703 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:19.098162 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:19.098167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:19.098379 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 06:48:19.098566 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 06:48:19.098853 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":149,"threads_started":1}
I20260619 06:48:19.100946 15564 heartbeater.cc:347] Connected to a master server at 127.14.116.254:42191
I20260619 06:48:19.101193 15564 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:19.101755 15564 heartbeater.cc:511] Master 127.14.116.254:42191 requested a full tablet report, sending...
I20260619 06:48:19.103166 15373 ts_manager.cc:194] Registered new tserver with Master: e02e2455210c4189b7ab53d6c2a3cdbf (127.14.116.193:45157)
I20260619 06:48:19.104322 15373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:59739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:19.113927 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:42191
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:19.219045 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:19.219316 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:19.219390 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:19.223304 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:19.223393 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:19.223485 15569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:19.228683 15569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:42191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:19.229678 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:19.230655 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:19.239679 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:19.239683 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:19.239921 15569 server_base.cc:1034] running on GCE node
W20260619 06:48:19.239760 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:19.240286 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:19.240819 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:19.241979 15569 hybrid_clock.cc:648] HybridClock initialized: now 1781851699241966 us; error 24 us; skew 500 ppm
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:19.244696 15569 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:19.245514 15569 webserver.cc:460] Webserver started at http://127.14.116.194:41741/ using document root <none> and password file <none>
I20260619 06:48:19.246039 15569 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:19.246130 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:19.246345 15569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:19.248296 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "94f39c7a6792448398166c3f0fe09eeb"
format_stamp: "Formatted at 2026-06-19 06:48:19 on dist-test-slave-9x3b"
server_key: "c3a99703950cc4e683cc6083c25615a4"
server_key_iv: "bec837cbbe1e8242a569f60236f95c8e"
server_key_version: "encryptionkey@0"
I20260619 06:48:19.248857 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "94f39c7a6792448398166c3f0fe09eeb"
format_stamp: "Formatted at 2026-06-19 06:48:19 on dist-test-slave-9x3b"
server_key: "c3a99703950cc4e683cc6083c25615a4"
server_key_iv: "bec837cbbe1e8242a569f60236f95c8e"
server_key_version: "encryptionkey@0"
I20260619 06:48:19.252250 15569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:48:19.254441 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:19.255602 15569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:19.255769 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "94f39c7a6792448398166c3f0fe09eeb"
format_stamp: "Formatted at 2026-06-19 06:48:19 on dist-test-slave-9x3b"
server_key: "c3a99703950cc4e683cc6083c25615a4"
server_key_iv: "bec837cbbe1e8242a569f60236f95c8e"
server_key_version: "encryptionkey@0"
I20260619 06:48:19.255894 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:19.265923 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:19.268643 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:19.268826 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:19.269351 15569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:19.270089 15569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:19.270134 15569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:19.270197 15569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:19.270234 15569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:19.278293 15569 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:44959
I20260619 06:48:19.278316 15697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:44959 every 8 connection(s)
I20260619 06:48:19.279109 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:19.279341 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 15569
I20260619 06:48:19.279465 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:19.279780 14803 external_mini_cluster.cc:1428] Setting key e983bd29bf26eecca9e64aa9e87c3f8e
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:48:19.292050 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.280934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:50363 (local address 127.14.116.254:42191)
0619 06:48:19.281090 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.281095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.282002 (+ 907us) server_negotiation.cc:407] Connection header received
0619 06:48:19.282860 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.282864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.282924 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.283018 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:19.284167 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.284680 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.285789 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.286212 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.286340 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.288594 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:19.288599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:19.288605 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:19.288638 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:19.290041 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.290546 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.290549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.290550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.290618 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.290891 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.290893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.290894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.291094 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:19.291197 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.291637 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.291787 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:48:19.292843 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.281402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:42191 (local address 127.14.116.194:50363)
0619 06:48:19.281860 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:19.281903 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:19.282663 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:19.283181 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:19.283187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:19.283521 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:19.284032 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.284045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.284803 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.284807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.285663 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.285672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.286354 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.286358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.286499 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.287062 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:19.287077 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:19.288436 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:19.290136 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:19.290141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:19.290150 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:19.290462 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:19.290692 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:19.290694 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:19.290695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:19.290810 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:19.291189 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:19.291194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:19.291398 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 06:48:19.291547 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:48:19.291888 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":132,"threads_started":1}
I20260619 06:48:19.294142 15698 heartbeater.cc:347] Connected to a master server at 127.14.116.254:42191
I20260619 06:48:19.294384 15698 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:19.294831 15698 heartbeater.cc:511] Master 127.14.116.254:42191 requested a full tablet report, sending...
I20260619 06:48:19.295842 15373 ts_manager.cc:194] Registered new tserver with Master: 94f39c7a6792448398166c3f0fe09eeb (127.14.116.194:44959)
I20260619 06:48:19.296701 15373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:50363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:19.304445 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:42191
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:19.412866 15703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:19.413120 15703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:19.413193 15703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:19.417098 15703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:19.417171 15703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:19.417264 15703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:19.422297 15703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:42191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:19.423244 15703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:19.424199 15703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:19.433024 15711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:19.433046 15709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:19.433019 15708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:19.433235 15703 server_base.cc:1034] running on GCE node
I20260619 06:48:19.433739 15703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:19.434351 15703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:19.435532 15703 hybrid_clock.cc:648] HybridClock initialized: now 1781851699435525 us; error 28 us; skew 500 ppm
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:19.438197 15703 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:19.438961 15703 webserver.cc:460] Webserver started at http://127.14.116.195:33127/ using document root <none> and password file <none>
I20260619 06:48:19.439489 15703 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:19.439574 15703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:19.439811 15703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:19.441701 15703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "7eb795e85b004c52a204cf73d591a822"
format_stamp: "Formatted at 2026-06-19 06:48:19 on dist-test-slave-9x3b"
server_key: "f77b962e42ccbbdbdf50160ec29bb057"
server_key_iv: "ce1483f079c496804a8ca4a15447902e"
server_key_version: "encryptionkey@0"
I20260619 06:48:19.442224 15703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "7eb795e85b004c52a204cf73d591a822"
format_stamp: "Formatted at 2026-06-19 06:48:19 on dist-test-slave-9x3b"
server_key: "f77b962e42ccbbdbdf50160ec29bb057"
server_key_iv: "ce1483f079c496804a8ca4a15447902e"
server_key_version: "encryptionkey@0"
I20260619 06:48:19.445973 15703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 06:48:19.448246 15718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:19.449357 15703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 06:48:19.449488 15703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "7eb795e85b004c52a204cf73d591a822"
format_stamp: "Formatted at 2026-06-19 06:48:19 on dist-test-slave-9x3b"
server_key: "f77b962e42ccbbdbdf50160ec29bb057"
server_key_iv: "ce1483f079c496804a8ca4a15447902e"
server_key_version: "encryptionkey@0"
I20260619 06:48:19.449589 15703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:19.462199 15703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:19.464911 15703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:19.465095 15703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:19.465638 15703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:19.466393 15703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:19.466461 15703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:19.466531 15703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:19.466583 15703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:19.474841 15703 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:37657
I20260619 06:48:19.474864 15831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:37657 every 8 connection(s)
I20260619 06:48:19.475668 15703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:19.479838 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 15703
I20260619 06:48:19.479957 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:19.480221 14803 external_mini_cluster.cc:1428] Setting key dd51bc0468e691f1f57a3c24e8b19a7d
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:19.487969 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.477319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:39771 (local address 127.14.116.254:42191)
0619 06:48:19.477487 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.477491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.478272 (+ 781us) server_negotiation.cc:407] Connection header received
0619 06:48:19.479067 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.479070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.479113 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.479185 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:19.480458 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.480999 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.482457 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.482662 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.482742 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.484993 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:19.484996 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:19.484998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:19.485021 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:19.486383 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.486780 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.486782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.486783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.486850 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.487107 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.487108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.487109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.487257 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:19.487326 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.487762 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.487849 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:48:19.488706 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.477703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:42191 (local address 127.14.116.195:39771)
0619 06:48:19.478116 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:19.478155 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:19.478868 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:19.479331 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:19.479337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:19.479741 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:19.480311 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.480332 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.481096 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.481100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.482355 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.482364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.482779 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.482782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.482919 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.483412 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:19.483427 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:19.484852 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:19.486471 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:19.486475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:19.486485 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:19.486704 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:19.486920 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:19.486922 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:19.486923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:19.487035 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:19.487335 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:19.487339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:19.487537 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 06:48:19.487731 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 06:48:19.487960 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260619 06:48:19.489741 15832 heartbeater.cc:347] Connected to a master server at 127.14.116.254:42191
I20260619 06:48:19.489964 15832 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:19.490392 15832 heartbeater.cc:511] Master 127.14.116.254:42191 requested a full tablet report, sending...
I20260619 06:48:19.491310 15373 ts_manager.cc:194] Registered new tserver with Master: 7eb795e85b004c52a204cf73d591a822 (127.14.116.195:37657)
I20260619 06:48:19.491875 15373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:39771
I20260619 06:48:19.493860 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:48:19.503870 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.497397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.14.116.254:42191)
0619 06:48:19.497514 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.497518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.497627 (+ 109us) server_negotiation.cc:407] Connection header received
0619 06:48:19.497714 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.497716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.497750 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.497812 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:19.498401 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.498863 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.499671 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.499888 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.499961 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.501297 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:19.501300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:19.501302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:19.501320 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:19.502533 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.502880 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.502882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.502883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.502949 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.503157 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.503159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.503160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.503341 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:19.503438 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.503672 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.503768 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:48:19.508131 15373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:19.510967 15373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:19.524781 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.518748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:45157 (local address 127.0.0.1:40542)
0619 06:48:19.520141 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:19.520155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:19.520277 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:19.520830 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:19.520835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:19.520849 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:19.520972 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.520979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.521959 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.521962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.523186 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.523195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.524232 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.524236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.524360 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.524403 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 06:48:19.524507 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:48:19.524619 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":47,"thread_start_us":74,"threads_started":1}
I20260619 06:48:19.524811 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.518867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.14.116.193:45157)
0619 06:48:19.519422 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.519434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.520221 (+ 787us) server_negotiation.cc:407] Connection header received
0619 06:48:19.520403 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.520408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.520565 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.520664 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:19.521093 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.521833 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.523303 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.524108 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.524235 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.524362 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.524556 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.524654 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":93,"threads_started":1}
I20260619 06:48:19.524957 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.519078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:44959 (local address 127.0.0.1:50804)
0619 06:48:19.519980 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:19.519992 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:19.520097 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:19.520736 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:19.520739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:19.520754 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:19.520861 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.520868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.522207 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.522211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.523420 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.523427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.524581 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.524584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.524669 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.524698 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:48:19.524765 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 06:48:19.524837 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":67,"thread_start_us":95,"threads_started":1}
I20260619 06:48:19.525395 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.519405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.14.116.194:44959)
0619 06:48:19.519832 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.519843 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.520082 (+ 239us) server_negotiation.cc:407] Connection header received
0619 06:48:19.520246 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.520251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.520441 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.520575 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:19.520987 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.522059 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.523701 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.524280 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.524402 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.524523 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.525162 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.525255 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":104,"threads_started":1}
I20260619 06:48:19.525403 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.519511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.14.116.195:37657)
0619 06:48:19.519908 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.519920 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.519944 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:48:19.520017 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.520022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.520221 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.520402 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:19.520657 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.521729 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.523534 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.524326 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.524442 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.524558 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.525227 (+ 669us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.525300 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":130,"threads_started":1}
I20260619 06:48:19.525802 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.518821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:37657 (local address 127.0.0.1:33030)
0619 06:48:19.519683 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:19.519732 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:19.519851 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:19.520390 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:19.520395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:19.520417 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:19.520551 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.520557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.521866 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.521871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.523417 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.523427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.524618 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.524622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.524707 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.524729 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:48:19.524798 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 06:48:19.525672 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":135,"thread_start_us":85,"threads_started":1}
I20260619 06:48:19.527922 15632 tablet_service.cc:1467] Processing CreateTablet for tablet 18259dd7a63d4991883839b53572b183 (DEFAULT_TABLE table=test-table [id=41f64743079e455bbfd0d269be8ce464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:19.528030 15498 tablet_service.cc:1467] Processing CreateTablet for tablet 18259dd7a63d4991883839b53572b183 (DEFAULT_TABLE table=test-table [id=41f64743079e455bbfd0d269be8ce464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:19.528175 15766 tablet_service.cc:1467] Processing CreateTablet for tablet 18259dd7a63d4991883839b53572b183 (DEFAULT_TABLE table=test-table [id=41f64743079e455bbfd0d269be8ce464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:19.528960 15498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18259dd7a63d4991883839b53572b183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:19.528981 15632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18259dd7a63d4991883839b53572b183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:19.528985 15766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18259dd7a63d4991883839b53572b183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:19.534452 15849 tablet_bootstrap.cc:492] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf: Bootstrap starting.
I20260619 06:48:19.535715 15851 tablet_bootstrap.cc:492] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822: Bootstrap starting.
I20260619 06:48:19.537499 15851 tablet_bootstrap.cc:654] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:19.537941 15849 tablet_bootstrap.cc:654] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:19.538744 15851 log.cc:824] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:19.538743 15849 log.cc:824] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:19.539332 15850 tablet_bootstrap.cc:492] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb: Bootstrap starting.
I20260619 06:48:19.540721 15851 tablet_bootstrap.cc:492] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822: No bootstrap required, opened a new log
I20260619 06:48:19.540732 15849 tablet_bootstrap.cc:492] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf: No bootstrap required, opened a new log
I20260619 06:48:19.540908 15851 ts_tablet_manager.cc:1397] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:48:19.540908 15849 ts_tablet_manager.cc:1397] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 06:48:19.541496 15850 tablet_bootstrap.cc:654] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:19.542135 15850 log.cc:824] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:19.543872 15850 tablet_bootstrap.cc:492] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb: No bootstrap required, opened a new log
I20260619 06:48:19.544059 15850 ts_tablet_manager.cc:1397] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:48:19.544723 15849 raft_consensus.cc:348] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } }
I20260619 06:48:19.544723 15851 raft_consensus.cc:348] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } }
I20260619 06:48:19.544991 15849 raft_consensus.cc:374] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:19.544991 15851 raft_consensus.cc:374] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:19.545048 15851 raft_consensus.cc:729] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb795e85b004c52a204cf73d591a822, State: Initialized, Role: FOLLOWER
I20260619 06:48:19.545048 15849 raft_consensus.cc:729] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02e2455210c4189b7ab53d6c2a3cdbf, State: Initialized, Role: FOLLOWER
I20260619 06:48:19.545521 15849 consensus_queue.cc:260] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } }
I20260619 06:48:19.545521 15851 consensus_queue.cc:260] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } }
I20260619 06:48:19.546458 15849 ts_tablet_manager.cc:1428] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:48:19.546458 15851 ts_tablet_manager.cc:1428] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:48:19.546825 15564 heartbeater.cc:503] Master 127.14.116.254:42191 was elected leader, sending a full tablet report...
I20260619 06:48:19.547138 15832 heartbeater.cc:503] Master 127.14.116.254:42191 was elected leader, sending a full tablet report...
I20260619 06:48:19.547437 15850 raft_consensus.cc:348] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } }
I20260619 06:48:19.547711 15850 raft_consensus.cc:374] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:19.547775 15850 raft_consensus.cc:729] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f39c7a6792448398166c3f0fe09eeb, State: Initialized, Role: FOLLOWER
I20260619 06:48:19.548192 15850 consensus_queue.cc:260] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } }
I20260619 06:48:19.548949 15698 heartbeater.cc:503] Master 127.14.116.254:42191 was elected leader, sending a full tablet report...
I20260619 06:48:19.549090 15850 ts_tablet_manager.cc:1428] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 06:48:19.586649 15565 tablet.cc:2369] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:48:19.726603 15833 tablet.cc:2369] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:19.751901 15857 raft_consensus.cc:482] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:19.752082 15857 raft_consensus.cc:504] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } }
I20260619 06:48:19.753206 15857 leader_election.cc:288] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e02e2455210c4189b7ab53d6c2a3cdbf (127.14.116.193:45157), 7eb795e85b004c52a204cf73d591a822 (127.14.116.195:37657)
I20260619 06:48:19.757258 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.753623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:35013 (local address 127.14.116.193:45157)
0619 06:48:19.753748 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.753753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.753770 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:48:19.753830 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.753832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.753873 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.753999 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:19.754333 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.755079 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.756289 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.756697 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.756832 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.756884 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.757077 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.757144 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:48:19.757262 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.753377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:45157 (local address 127.14.116.194:35013)
0619 06:48:19.753510 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:19.753543 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:19.753643 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:19.754013 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:19.754017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:19.754037 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:19.754197 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.754204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.755191 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.755195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.756169 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.756178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.756832 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.756836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.756956 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.756973 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 06:48:19.757061 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:48:19.757129 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":20,"spinlock_wait_cycles":28544}
I20260619 06:48:19.757560 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.753508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:37657 (local address 127.14.116.194:49963)
0619 06:48:19.753944 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:19.753958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:19.754102 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:19.754473 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:19.754476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:19.754500 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:19.754590 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.754594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.755462 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.755466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.756457 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:19.756466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.757113 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.757117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:19.757212 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.757233 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:48:19.757326 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:48:19.757424 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":1,"spinlock_wait_cycles":2304,"thread_start_us":117,"threads_started":1}
I20260619 06:48:19.757776 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.753623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:49963 (local address 127.14.116.195:37657)
0619 06:48:19.753748 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.753753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.754027 (+ 274us) server_negotiation.cc:407] Connection header received
0619 06:48:19.754232 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.754236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.754284 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.754357 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:19.754702 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.755352 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.756582 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.756996 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.757122 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.757168 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.757609 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.757689 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:48:19.757874 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18259dd7a63d4991883839b53572b183" candidate_uuid: "94f39c7a6792448398166c3f0fe09eeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" is_pre_election: true
I20260619 06:48:19.758170 15518 raft_consensus.cc:2454] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f39c7a6792448398166c3f0fe09eeb in term 0.
I20260619 06:48:19.758270 15786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18259dd7a63d4991883839b53572b183" candidate_uuid: "94f39c7a6792448398166c3f0fe09eeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb795e85b004c52a204cf73d591a822" is_pre_election: true
I20260619 06:48:19.758560 15786 raft_consensus.cc:2454] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f39c7a6792448398166c3f0fe09eeb in term 0.
I20260619 06:48:19.758633 15586 leader_election.cc:302] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f39c7a6792448398166c3f0fe09eeb, e02e2455210c4189b7ab53d6c2a3cdbf; no voters:
I20260619 06:48:19.758955 15857 raft_consensus.cc:2781] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:19.759037 15857 raft_consensus.cc:482] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:19.759091 15857 raft_consensus.cc:3037] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:19.760185 15857 raft_consensus.cc:504] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } }
I20260619 06:48:19.760492 15857 leader_election.cc:288] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [CANDIDATE]: Term 1 election: Requested vote from peers e02e2455210c4189b7ab53d6c2a3cdbf (127.14.116.193:45157), 7eb795e85b004c52a204cf73d591a822 (127.14.116.195:37657)
I20260619 06:48:19.760782 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18259dd7a63d4991883839b53572b183" candidate_uuid: "94f39c7a6792448398166c3f0fe09eeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf"
I20260619 06:48:19.760854 15786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18259dd7a63d4991883839b53572b183" candidate_uuid: "94f39c7a6792448398166c3f0fe09eeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb795e85b004c52a204cf73d591a822"
I20260619 06:48:19.760890 15518 raft_consensus.cc:3037] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:19.760988 15786 raft_consensus.cc:3037] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:19.761857 15518 raft_consensus.cc:2454] T 18259dd7a63d4991883839b53572b183 P e02e2455210c4189b7ab53d6c2a3cdbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f39c7a6792448398166c3f0fe09eeb in term 1.
I20260619 06:48:19.762003 15786 raft_consensus.cc:2454] T 18259dd7a63d4991883839b53572b183 P 7eb795e85b004c52a204cf73d591a822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f39c7a6792448398166c3f0fe09eeb in term 1.
I20260619 06:48:19.762172 15586 leader_election.cc:302] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f39c7a6792448398166c3f0fe09eeb, e02e2455210c4189b7ab53d6c2a3cdbf; no voters:
I20260619 06:48:19.762351 15857 raft_consensus.cc:2781] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:19.762542 15857 raft_consensus.cc:686] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [term 1 LEADER]: Becoming Leader. State: Replica: 94f39c7a6792448398166c3f0fe09eeb, State: Running, Role: LEADER
I20260619 06:48:19.762844 15857 consensus_queue.cc:237] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } }
I20260619 06:48:19.765798 15373 catalog_manager.cc:5495] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f39c7a6792448398166c3f0fe09eeb (127.14.116.194). New cstate: current_term: 1 leader_uuid: "94f39c7a6792448398166c3f0fe09eeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e02e2455210c4189b7ab53d6c2a3cdbf" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb795e85b004c52a204cf73d591a822" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 37657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94f39c7a6792448398166c3f0fe09eeb" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 44959 } health_report { overall_health: HEALTHY } } }
W20260619 06:48:19.780251 15699 tablet.cc:2369] T 18259dd7a63d4991883839b53572b183 P 94f39c7a6792448398166c3f0fe09eeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:19.821102 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.813143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.14.116.193:45157)
0619 06:48:19.813300 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.813304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.813363 (+ 59us) server_negotiation.cc:407] Connection header received
0619 06:48:19.813463 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.813466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.813505 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.813571 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:19.814202 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.814688 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.815444 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.815697 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.815769 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.818122 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:19.818134 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:19.818148 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:19.818169 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:19.819573 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.820010 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.820015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.820017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.820094 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.820315 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.820316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.820318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.820530 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:19.820680 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.820801 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.820886 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:19.848824 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.841049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.14.116.193:45157)
0619 06:48:19.841179 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.841182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.841340 (+ 158us) server_negotiation.cc:407] Connection header received
0619 06:48:19.841446 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.841448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.841485 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.841552 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:19.842147 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.842645 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.843473 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.843743 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.843820 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.846052 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:19.846055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:19.846057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:19.846077 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:19.847492 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.847886 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.847889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.847890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.847960 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.848173 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.848175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.848176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.848317 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:19.848409 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.848627 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.848713 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:19.872599 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:19.864970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.14.116.193:45157)
0619 06:48:19.865126 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:19.865131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:19.865280 (+ 149us) server_negotiation.cc:407] Connection header received
0619 06:48:19.865448 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:19.865451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:19.865499 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:19.865584 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:19.866217 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.866721 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.867542 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:19.867911 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:19.868023 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:19.870162 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:19.870165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:19.870168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:19.870188 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:19.871326 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.871715 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.871718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.871719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.871785 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:19.872002 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:19.872004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:19.872005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:19.872148 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:19.872240 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:19.872404 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 06:48:19.872485 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 06:48:19.873951 15478 tablet_service.cc:2908] Rejecting scan request for tablet 18259dd7a63d4991883839b53572b183: Uninitialized: clean time has not yet been initialized
I20260619 06:48:19.875850 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 15435
I20260619 06:48:19.881039 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 15569
I20260619 06:48:19.886309 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 15703
I20260619 06:48:19.891471 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 15341
2026-06-19T06:48:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2377 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15873](info): set up 2 sockets
Jun 19 06:48:19 dist-test-slave-9x3b krb5kdc[15873](info): commencing operation
krb5kdc: starting...
W20260619 06:48:20.931329 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:21.042484 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:44269
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:44269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:21.150534 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:21.150799 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:21.150868 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:21.154759 15886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:21.154839 15886 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:48:21.154877 15886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:21.154940 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:21.154978 15886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:21.155014 15886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:21.160125 15886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:44269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:44269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:21.161082 15886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:21.162016 15886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:48:21.170054 15886 server_base.cc:1034] running on GCE node
W20260619 06:48:21.170140 15891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:21.170083 15894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:21.170075 15892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:21.171013 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:21.172108 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:21.173291 15886 hybrid_clock.cc:648] HybridClock initialized: now 1781851701173281 us; error 33 us; skew 500 ppm
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:21.178436 15886 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:21.179235 15886 webserver.cc:460] Webserver started at http://127.14.116.254:38749/ using document root <none> and password file <none>
I20260619 06:48:21.179816 15886 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:21.179903 15886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:21.180130 15886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:21.182029 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "6ff6262ce08a440db109aaa995d9a307"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "9c335b9a5536f9c1b1c49660216e870d"
server_key_iv: "28af0fa2856a407e32c25b08f57c5047"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.182565 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "6ff6262ce08a440db109aaa995d9a307"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "9c335b9a5536f9c1b1c49660216e870d"
server_key_iv: "28af0fa2856a407e32c25b08f57c5047"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.186034 15886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 06:48:21.188282 15901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:21.189416 15886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:21.189549 15886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "6ff6262ce08a440db109aaa995d9a307"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "9c335b9a5536f9c1b1c49660216e870d"
server_key_iv: "28af0fa2856a407e32c25b08f57c5047"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.189651 15886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:21.214561 15886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:21.217283 15886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:21.217476 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:21.224920 15886 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:44269
I20260619 06:48:21.224941 15963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:44269 every 8 connection(s)
I20260619 06:48:21.225801 15886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:21.228015 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 15886
I20260619 06:48:21.228129 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:21.228405 14803 external_mini_cluster.cc:1428] Setting key b61971b07f1cd3eb9beebc4a0b44ad27
I20260619 06:48:21.228524 15964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:21.234418 15964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307: Bootstrap starting.
I20260619 06:48:21.236801 15964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307: Neither blocks nor log segments found. Creating new log.
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:21.237663 15964 log.cc:824] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:21.239382 15964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307: No bootstrap required, opened a new log
I20260619 06:48:21.242311 15964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff6262ce08a440db109aaa995d9a307" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44269 } }
I20260619 06:48:21.242518 15964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:21.242586 15964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6262ce08a440db109aaa995d9a307, State: Initialized, Role: FOLLOWER
I20260619 06:48:21.242903 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.229704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.14.116.254:44269)
0619 06:48:21.230114 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.230129 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.230162 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:48:21.231074 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.231091 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.231397 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.231737 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:21.232504 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.233489 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.234350 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.234859 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.235035 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.237667 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:21.237672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:21.237683 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:21.237708 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:21.239998 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.240438 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.240450 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.240454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.240572 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.240857 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.240861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.240863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.241213 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:21.241480 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.241774 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.242071 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":149,"threads_started":1}
I20260619 06:48:21.243014 15964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff6262ce08a440db109aaa995d9a307" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44269 } }
I20260619 06:48:21.243216 15964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:21.243309 15964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:21.243436 15964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:21.244299 15964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff6262ce08a440db109aaa995d9a307" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44269 } }
I20260619 06:48:21.244619 15964 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ff6262ce08a440db109aaa995d9a307; no voters:
I20260619 06:48:21.244935 15964 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:21.245040 15969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:21.245262 15969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff6262ce08a440db109aaa995d9a307, State: Running, Role: LEADER
I20260619 06:48:21.245573 15969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff6262ce08a440db109aaa995d9a307" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44269 } }
I20260619 06:48:21.246035 15964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:21.247023 15971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff6262ce08a440db109aaa995d9a307. Latest consensus state: current_term: 1 leader_uuid: "6ff6262ce08a440db109aaa995d9a307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff6262ce08a440db109aaa995d9a307" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44269 } } }
I20260619 06:48:21.247267 15970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ff6262ce08a440db109aaa995d9a307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff6262ce08a440db109aaa995d9a307" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44269 } } }
I20260619 06:48:21.247368 15971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:21.247449 15970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:21.247860 15979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:21.250313 15979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:21.255434 15979 catalog_manager.cc:1347] Generated new cluster ID: 56662cc6810a45ed85c49da106745103
I20260619 06:48:21.255493 15979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:21.262980 15979 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:21.264101 15979 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:21.272601 15979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307: Generated new TSK 0
I20260619 06:48:21.273320 15979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:21.304023 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44269
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:21.409924 15990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:21.410151 15990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:21.410192 15990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:21.414170 15990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:21.414235 15990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:21.414273 15990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:48:21.414361 15990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:21.419361 15990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.254:44269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:21.420281 15990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:21.421203 15990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:21.430040 15995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:21.430138 15998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:21.430178 15996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:21.430259 15990 server_base.cc:1034] running on GCE node
I20260619 06:48:21.430699 15990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:21.431221 15990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:21.432411 15990 hybrid_clock.cc:648] HybridClock initialized: now 1781851701432400 us; error 33 us; skew 500 ppm
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:21.434934 15990 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:21.435770 15990 webserver.cc:460] Webserver started at http://127.14.116.193:41769/ using document root <none> and password file <none>
I20260619 06:48:21.436349 15990 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:21.436442 15990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:21.436658 15990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:21.438614 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "435effe2c7ee434cb8117bb10f44e029"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "ace758cc59af492af4a359ca2e400618"
server_key_iv: "0629d3bc5df8a5e6572a316db706c912"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.439150 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "435effe2c7ee434cb8117bb10f44e029"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "ace758cc59af492af4a359ca2e400618"
server_key_iv: "0629d3bc5df8a5e6572a316db706c912"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.442518 15990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:48:21.444734 16005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:21.445838 15990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:21.445976 15990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "435effe2c7ee434cb8117bb10f44e029"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "ace758cc59af492af4a359ca2e400618"
server_key_iv: "0629d3bc5df8a5e6572a316db706c912"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.446079 15990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:21.458659 15990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:21.461169 15990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:21.461350 15990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:21.462598 15990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:21.462670 15990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:21.462739 15990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:21.462783 15990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:21.473531 15990 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:46361
I20260619 06:48:21.473542 16120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:46361 every 8 connection(s)
I20260619 06:48:21.474660 15990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:21.476691 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.463372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:38639 (local address 127.14.116.254:44269)
0619 06:48:21.463500 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.463504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.464460 (+ 956us) server_negotiation.cc:407] Connection header received
0619 06:48:21.465474 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.465480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.465541 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.465601 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:21.467199 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.468136 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.469810 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.470242 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.470364 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.473138 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:21.473145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:21.473150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:21.473181 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:21.474844 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.475390 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.475396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.475401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.475493 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.475852 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.475854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.475855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.476013 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:21.476088 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.476483 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.476577 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:48:21.477397 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.463884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44269 (local address 127.14.116.193:38639)
0619 06:48:21.464319 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:21.464356 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:21.465232 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:21.466037 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:21.466045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:21.466498 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:21.467047 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.467063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.468270 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.468273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.469677 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.469688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.470398 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.470402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.470577 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.471129 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:21.471143 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:21.472973 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:21.474957 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.474961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.474990 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.475293 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.475604 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.475646 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.475651 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.475784 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.476097 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:21.476101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:21.476287 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 06:48:21.476421 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 06:48:21.476638 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":127,"threads_started":1}
I20260619 06:48:21.478497 16121 heartbeater.cc:347] Connected to a master server at 127.14.116.254:44269
I20260619 06:48:21.478641 16121 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:21.479099 16121 heartbeater.cc:511] Master 127.14.116.254:44269 requested a full tablet report, sending...
I20260619 06:48:21.479302 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 15990
I20260619 06:48:21.479415 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:21.479718 14803 external_mini_cluster.cc:1428] Setting key 86cd72e673856300de8973e0046a2c32
I20260619 06:48:21.480755 15917 ts_manager.cc:194] Registered new tserver with Master: 435effe2c7ee434cb8117bb10f44e029 (127.14.116.193:46361)
I20260619 06:48:21.482088 15917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:38639
I20260619 06:48:21.488000 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.480844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:33917 (local address 127.14.116.254:44269)
0619 06:48:21.480956 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.480960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.480973 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:48:21.481092 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.481096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.481134 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.481212 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:21.481676 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.482399 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.483552 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.483867 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.483951 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.485060 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:21.485063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:21.485064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:21.485083 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:21.486416 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.486854 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.486859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.486861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.486939 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.487200 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.487203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.487204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.487374 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:21.487477 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.487747 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.487845 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:48:21.488011 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.480620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44269 (local address 127.14.116.193:33917)
0619 06:48:21.480893 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:21.480907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:21.480991 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:21.481246 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:21.481250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:21.481447 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:21.481554 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.481561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.482533 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.482539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.483427 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.483436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.483972 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.483976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.484062 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.484476 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:21.484492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:21.484916 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:21.486546 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.486549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.486551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.486760 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.487025 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.487029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.487031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.487108 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.487470 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:21.487473 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:21.487561 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 06:48:21.487697 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 06:48:21.487827 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:21.504885 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44269
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:21.610909 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:21.611153 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:21.611195 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:21.615149 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:21.615200 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:21.615269 16130 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:48:21.615351 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:21.620409 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.254:44269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:21.621335 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:21.622290 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:21.631137 16136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:21.631145 16135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:21.631146 16138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:21.631475 16130 server_base.cc:1034] running on GCE node
I20260619 06:48:21.631913 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:21.632441 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:21.633592 16130 hybrid_clock.cc:648] HybridClock initialized: now 1781851701633581 us; error 23 us; skew 500 ppm
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:21.636214 16130 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:21.637082 16130 webserver.cc:460] Webserver started at http://127.14.116.194:41883/ using document root <none> and password file <none>
I20260619 06:48:21.637620 16130 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:21.637681 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:21.637921 16130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:21.639964 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "a068558fbb8f4638ac9bfbff3265411f"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "fca374ee6fbcee67f3ce083c289b2a5f"
server_key_iv: "54889756147ae4407eac8a3482faebd9"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.640517 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "a068558fbb8f4638ac9bfbff3265411f"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "fca374ee6fbcee67f3ce083c289b2a5f"
server_key_iv: "54889756147ae4407eac8a3482faebd9"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.644063 16130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:48:21.646406 16145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:21.647766 16130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:21.647882 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "a068558fbb8f4638ac9bfbff3265411f"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "fca374ee6fbcee67f3ce083c289b2a5f"
server_key_iv: "54889756147ae4407eac8a3482faebd9"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.647991 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:21.674503 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:21.677183 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:21.677381 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:21.678813 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:21.678875 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:21.678965 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:21.678998 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:21.689563 16130 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:42085
I20260619 06:48:21.689653 16260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:42085 every 8 connection(s)
I20260619 06:48:21.690519 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:21.693895 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.679542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:54891 (local address 127.14.116.254:44269)
0619 06:48:21.679696 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.679700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.681374 (+ 1674us) server_negotiation.cc:407] Connection header received
0619 06:48:21.682662 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.682666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.682726 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.682817 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:21.684504 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.685146 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.686908 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.687126 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.687206 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.690194 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:21.690198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:21.690201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:21.690234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:21.691801 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.692321 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.692324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.692326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.692416 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.692805 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.692808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.692810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.692994 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:21.693094 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.693583 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.693705 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:48:21.694628 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.680651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44269 (local address 127.14.116.194:54891)
0619 06:48:21.681211 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:21.681252 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:21.682414 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:21.683032 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:21.683043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:21.683565 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:21.684315 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.684337 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.685263 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.685268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.686771 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.686783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.687278 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.687283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.687507 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.688199 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:21.688225 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:21.689999 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:21.691914 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.691921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.691937 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.692207 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.692509 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.692513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.692515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.692705 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.693124 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:21.693130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:21.693353 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 06:48:21.693530 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 06:48:21.693835 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":147,"threads_started":1}
I20260619 06:48:21.695735 16261 heartbeater.cc:347] Connected to a master server at 127.14.116.254:44269
I20260619 06:48:21.695885 16261 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:21.696341 16261 heartbeater.cc:511] Master 127.14.116.254:44269 requested a full tablet report, sending...
I20260619 06:48:21.697299 15917 ts_manager.cc:194] Registered new tserver with Master: a068558fbb8f4638ac9bfbff3265411f (127.14.116.194:42085)
I20260619 06:48:21.697886 15917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:54891
I20260619 06:48:21.700496 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 16130
I20260619 06:48:21.700618 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:21.700927 14803 external_mini_cluster.cc:1428] Setting key d6895ec44596c44dd9e4221602b10075
I20260619 06:48:21.705073 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.697913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:51425 (local address 127.14.116.254:44269)
0619 06:48:21.698070 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.698074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.698251 (+ 177us) server_negotiation.cc:407] Connection header received
0619 06:48:21.698351 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.698353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.698413 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.698490 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:21.698878 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.699587 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.700608 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.700864 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.700942 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.702052 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:21.702055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:21.702060 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:21.702083 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:21.703470 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.703919 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.703925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.703927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.704001 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.704265 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.704269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.704271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.704468 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:21.704574 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.704821 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.704924 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 06:48:21.705087 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.697847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44269 (local address 127.14.116.194:51425)
0619 06:48:21.698166 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:21.698179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:21.698272 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:21.698512 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:21.698514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:21.698669 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:21.698776 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.698780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.699755 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.699759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.700480 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.700491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.700979 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.700983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.701080 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.701567 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:21.701583 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:21.701918 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:21.703590 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.703593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.703595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.703816 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.704097 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.704100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.704102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.704181 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.704565 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:21.704568 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:21.704660 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 06:48:21.704759 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:48:21.704899 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:21.724265 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44269
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:21.829681 16270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:21.829918 16270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:21.829965 16270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:21.833820 16270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:21.833865 16270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:21.833902 16270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:48:21.833941 16270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:21.839013 16270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.254:44269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:21.839942 16270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:21.840811 16270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:21.849740 16276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:21.849771 16275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:21.849777 16278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:21.850015 16270 server_base.cc:1034] running on GCE node
I20260619 06:48:21.850548 16270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:21.851162 16270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:21.852312 16270 hybrid_clock.cc:648] HybridClock initialized: now 1781851701852279 us; error 46 us; skew 500 ppm
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:21.854784 16270 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:21.855554 16270 webserver.cc:460] Webserver started at http://127.14.116.195:34679/ using document root <none> and password file <none>
I20260619 06:48:21.856076 16270 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:21.856133 16270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:21.856277 16270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:21.858182 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "3d7fc86bf79447ff8d354d47e6eded95"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "1b9e4d7a8b347166c660e3f3495d1606"
server_key_iv: "400a0e570c675370eaeb56d40a5c5225"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.858680 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "3d7fc86bf79447ff8d354d47e6eded95"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "1b9e4d7a8b347166c660e3f3495d1606"
server_key_iv: "400a0e570c675370eaeb56d40a5c5225"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.862049 16270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 06:48:21.864212 16285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:21.865272 16270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:21.865401 16270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "3d7fc86bf79447ff8d354d47e6eded95"
format_stamp: "Formatted at 2026-06-19 06:48:21 on dist-test-slave-9x3b"
server_key: "1b9e4d7a8b347166c660e3f3495d1606"
server_key_iv: "400a0e570c675370eaeb56d40a5c5225"
server_key_version: "encryptionkey@0"
I20260619 06:48:21.865517 16270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:21.878575 16270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:21.881354 16270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:21.881549 16270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:21.882921 16270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:21.882992 16270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:21.883069 16270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:21.883091 16270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:21.893867 16270 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:35657
I20260619 06:48:21.893987 16400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:35657 every 8 connection(s)
I20260619 06:48:21.894836 16270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:21.896886 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.883533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:49261 (local address 127.14.116.254:44269)
0619 06:48:21.883680 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.883685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.885243 (+ 1558us) server_negotiation.cc:407] Connection header received
0619 06:48:21.886405 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.886408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.886465 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.886556 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:21.888191 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.888776 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.890058 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.890290 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.890402 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.893123 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:21.893126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:21.893129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:21.893152 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:21.894654 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.895227 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.895231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.895233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.895321 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.895716 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.895720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.895722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.895989 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:21.896130 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.896609 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.896723 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:48:21.897698 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.884587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44269 (local address 127.14.116.195:49261)
0619 06:48:21.885085 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:21.885145 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:21.886195 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:21.886752 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:21.886761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:21.887249 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:21.888011 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.888032 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.888900 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.888903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.889936 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.889945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.890449 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.890453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.890615 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.891152 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:21.891167 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:21.892967 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:21.894773 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.894780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.894794 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.895109 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.895439 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.895442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.895444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.895595 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.896145 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:21.896153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:21.896377 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 06:48:21.896554 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 06:48:21.896850 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":143,"threads_started":1}
I20260619 06:48:21.898710 16401 heartbeater.cc:347] Connected to a master server at 127.14.116.254:44269
I20260619 06:48:21.898834 16401 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:21.899344 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 16270
I20260619 06:48:21.899339 16401 heartbeater.cc:511] Master 127.14.116.254:44269 requested a full tablet report, sending...
I20260619 06:48:21.899457 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:21.899793 14803 external_mini_cluster.cc:1428] Setting key 31b46750a11e5b4cec4ac9d963773c2c
I20260619 06:48:21.900410 15918 ts_manager.cc:194] Registered new tserver with Master: 3d7fc86bf79447ff8d354d47e6eded95 (127.14.116.195:35657)
I20260619 06:48:21.900981 15918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:49261
I20260619 06:48:21.902333 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:48:21.909078 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.900957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44269 (local address 127.14.116.195:50369)
0619 06:48:21.901295 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:21.901310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:21.901398 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:21.901697 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:21.901699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:21.901915 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:21.902059 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.902070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.903084 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.903091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.904101 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.904115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.904735 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.904738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.904805 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.905265 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:21.905278 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:21.905565 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:21.907511 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.907515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.907517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.907795 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.908058 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:21.908065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:21.908070 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:21.908173 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:21.908507 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:21.908513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:21.908628 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 06:48:21.908745 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 06:48:21.908920 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260619 06:48:21.909143 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.900991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:50369 (local address 127.14.116.254:44269)
0619 06:48:21.901106 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.901110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.901388 (+ 278us) server_negotiation.cc:407] Connection header received
0619 06:48:21.901475 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.901481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.901524 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.901598 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:21.902184 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.902913 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.904234 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.904620 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.904736 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.905698 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:21.905705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:21.905710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:21.905739 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:21.907397 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.907890 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.907893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.907894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.907964 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.908267 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.908269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.908270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.908418 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:21.908489 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.908825 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.908948 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:48:21 dist-test-slave-9x3b krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:21.927405 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.920278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.14.116.254:44269)
0619 06:48:21.920411 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.920414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.920426 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:48:21.920456 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.920459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.920490 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.920555 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:21.921186 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.921670 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.922418 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.922611 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.922680 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.924874 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:21.924877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:21.924879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:21.924899 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:21.926052 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.926422 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.926424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.926425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.926490 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:21.926698 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:21.926700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:21.926701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:21.926842 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:21.926936 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.927118 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.927196 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:48:21.929909 15918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:21.932791 15918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:21.946427 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.940476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46361 (local address 127.0.0.1:37858)
0619 06:48:21.941160 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:21.941210 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:21.941342 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:21.942113 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:21.942118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:21.942142 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:21.942271 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.942278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.943305 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.943311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.944765 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.944775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.945885 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.945888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.946002 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.946035 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:48:21.946147 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:48:21.946255 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":28,"spinlock_wait_cycles":6656,"thread_start_us":73,"threads_started":1}
I20260619 06:48:21.946439 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.941148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.14.116.193:46361)
0619 06:48:21.941619 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.941632 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.941655 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:48:21.941728 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.941735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.941957 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.942111 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:21.942390 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.943200 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.944889 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.945760 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.945908 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.946050 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.946173 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.946254 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":107,"threads_started":1}
I20260619 06:48:21.946995 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.940725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:35657 (local address 127.0.0.1:48288)
0619 06:48:21.941575 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:21.941587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:21.941670 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:21.942392 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:21.942395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:21.942410 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:21.942503 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.942508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.943881 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.943883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.945278 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.945288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.946568 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.946572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.946735 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.946756 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:48:21.946830 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:48:21.946885 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":46,"thread_start_us":85,"threads_started":1}
I20260619 06:48:21.947005 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.940532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.14.116.194:42085)
0619 06:48:21.941754 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.941771 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.941816 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:48:21.941886 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.941893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.942093 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.942235 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:21.942524 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.943723 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.945067 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.945854 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.946006 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.946116 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.946771 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.946866 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116,"thread_start_us":109,"threads_started":1}
I20260619 06:48:21.947137 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.941034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.14.116.195:35657)
0619 06:48:21.941938 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:21.941948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:21.941970 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:48:21.942037 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:21.942043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:21.942208 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:21.942341 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:21.942618 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.943745 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.945447 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.946249 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.946384 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.946512 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:21.946886 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 06:48:21.946988 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":93,"threads_started":1}
I20260619 06:48:21.946633 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:21.940375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:42085 (local address 127.0.0.1:59368)
0619 06:48:21.941160 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:21.941210 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:21.941341 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:21.942268 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:21.942273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:21.942297 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:21.942392 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.942399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.943862 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.943866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.944964 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:21.944971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:21.946179 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:21.946182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:21.946266 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:21.946304 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:48:21.946369 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 06:48:21.946479 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":3,"spinlock_wait_cycles":3712,"thread_start_us":74,"threads_started":1}
I20260619 06:48:21.949883 16195 tablet_service.cc:1467] Processing CreateTablet for tablet c6b0f95809434c3e898070fcaf3882ae (DEFAULT_TABLE table=test-table [id=c81aec797a4f43eda9423c524104cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:21.949896 16055 tablet_service.cc:1467] Processing CreateTablet for tablet c6b0f95809434c3e898070fcaf3882ae (DEFAULT_TABLE table=test-table [id=c81aec797a4f43eda9423c524104cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:21.949901 16335 tablet_service.cc:1467] Processing CreateTablet for tablet c6b0f95809434c3e898070fcaf3882ae (DEFAULT_TABLE table=test-table [id=c81aec797a4f43eda9423c524104cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:21.950839 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b0f95809434c3e898070fcaf3882ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:21.950872 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b0f95809434c3e898070fcaf3882ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:21.950891 16335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b0f95809434c3e898070fcaf3882ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:21.957269 16424 tablet_bootstrap.cc:492] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: Bootstrap starting.
I20260619 06:48:21.958719 16425 tablet_bootstrap.cc:492] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: Bootstrap starting.
I20260619 06:48:21.959345 16424 tablet_bootstrap.cc:654] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:21.959919 16426 tablet_bootstrap.cc:492] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: Bootstrap starting.
I20260619 06:48:21.960157 16424 log.cc:824] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:21.960665 16425 tablet_bootstrap.cc:654] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:21.961477 16425 log.cc:824] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:21.961920 16426 tablet_bootstrap.cc:654] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:21.962081 16424 tablet_bootstrap.cc:492] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: No bootstrap required, opened a new log
I20260619 06:48:21.962292 16424 ts_tablet_manager.cc:1397] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:48:21.962663 16426 log.cc:824] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:21.963393 16425 tablet_bootstrap.cc:492] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: No bootstrap required, opened a new log
I20260619 06:48:21.963582 16425 ts_tablet_manager.cc:1397] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:48:21.964286 16426 tablet_bootstrap.cc:492] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: No bootstrap required, opened a new log
I20260619 06:48:21.964473 16426 ts_tablet_manager.cc:1397] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:48:21.966179 16424 raft_consensus.cc:348] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:21.966441 16424 raft_consensus.cc:374] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:21.966526 16424 raft_consensus.cc:729] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068558fbb8f4638ac9bfbff3265411f, State: Initialized, Role: FOLLOWER
I20260619 06:48:21.966987 16424 consensus_queue.cc:260] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:21.967258 16426 raft_consensus.cc:348] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:21.967530 16426 raft_consensus.cc:374] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:21.967628 16426 raft_consensus.cc:729] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7fc86bf79447ff8d354d47e6eded95, State: Initialized, Role: FOLLOWER
I20260619 06:48:21.967280 16425 raft_consensus.cc:348] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:21.967692 16261 heartbeater.cc:503] Master 127.14.116.254:44269 was elected leader, sending a full tablet report...
I20260619 06:48:21.967773 16425 raft_consensus.cc:374] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:21.967985 16424 ts_tablet_manager.cc:1428] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 06:48:21.968088 16426 consensus_queue.cc:260] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:21.967851 16425 raft_consensus.cc:729] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435effe2c7ee434cb8117bb10f44e029, State: Initialized, Role: FOLLOWER
I20260619 06:48:21.968660 16401 heartbeater.cc:503] Master 127.14.116.254:44269 was elected leader, sending a full tablet report...
I20260619 06:48:21.969012 16426 ts_tablet_manager.cc:1428] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 06:48:21.969048 16425 consensus_queue.cc:260] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:21.969671 16121 heartbeater.cc:503] Master 127.14.116.254:44269 was elected leader, sending a full tablet report...
I20260619 06:48:21.969951 16425 ts_tablet_manager.cc:1428] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 06:48:21.975769 16122 tablet.cc:2369] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:22.093756 16431 raft_consensus.cc:482] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:22.093986 16431 raft_consensus.cc:504] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.095049 16431 leader_election.cc:288] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a068558fbb8f4638ac9bfbff3265411f (127.14.116.194:42085), 435effe2c7ee434cb8117bb10f44e029 (127.14.116.193:46361)
I20260619 06:48:22.098802 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.095289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:42085 (local address 127.14.116.195:43173)
0619 06:48:22.095404 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.095421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.095506 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.095813 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.095816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.095838 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:22.095977 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.095986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.096855 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.096861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.097845 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.097855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.098464 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.098470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.098580 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.098599 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.098656 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.098706 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 06:48:22.098891 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.095394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:43173 (local address 127.14.116.194:42085)
0619 06:48:22.095542 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.095548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.095564 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:48:22.095639 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.095643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.095698 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.095779 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:22.096093 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.096761 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.097977 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.098354 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.098483 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.098533 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.098688 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.098753 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:48:22.099639 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.095360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:43779 (local address 127.14.116.193:46361)
0619 06:48:22.095519 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.095523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.096024 (+ 501us) server_negotiation.cc:407] Connection header received
0619 06:48:22.096126 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.096128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.096175 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.096251 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:22.096927 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.097390 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.098517 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.099087 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.099221 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.099266 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.099449 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.099509 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 06:48:22.099648 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.095289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46361 (local address 127.14.116.195:43779)
0619 06:48:22.095917 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.095933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.096039 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.096263 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.096265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.096287 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:22.096790 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.096798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.097493 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.097497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.098349 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.098356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.099219 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.099222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.099318 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.099343 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.099429 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.099492 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":289,"thread_start_us":108,"threads_started":1}
I20260619 06:48:22.099655 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b0f95809434c3e898070fcaf3882ae" candidate_uuid: "3d7fc86bf79447ff8d354d47e6eded95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068558fbb8f4638ac9bfbff3265411f" is_pre_election: true
I20260619 06:48:22.099992 16215 raft_consensus.cc:2454] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d7fc86bf79447ff8d354d47e6eded95 in term 0.
I20260619 06:48:22.100159 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b0f95809434c3e898070fcaf3882ae" candidate_uuid: "3d7fc86bf79447ff8d354d47e6eded95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435effe2c7ee434cb8117bb10f44e029" is_pre_election: true
I20260619 06:48:22.100442 16073 raft_consensus.cc:2454] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d7fc86bf79447ff8d354d47e6eded95 in term 0.
I20260619 06:48:22.100580 16288 leader_election.cc:302] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7fc86bf79447ff8d354d47e6eded95, a068558fbb8f4638ac9bfbff3265411f; no voters:
I20260619 06:48:22.100858 16431 raft_consensus.cc:2781] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:22.100957 16431 raft_consensus.cc:482] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:22.100998 16431 raft_consensus.cc:3037] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:22.102205 16431 raft_consensus.cc:504] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.102622 16431 leader_election.cc:288] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [CANDIDATE]: Term 1 election: Requested vote from peers a068558fbb8f4638ac9bfbff3265411f (127.14.116.194:42085), 435effe2c7ee434cb8117bb10f44e029 (127.14.116.193:46361)
I20260619 06:48:22.102907 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b0f95809434c3e898070fcaf3882ae" candidate_uuid: "3d7fc86bf79447ff8d354d47e6eded95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435effe2c7ee434cb8117bb10f44e029"
I20260619 06:48:22.103044 16073 raft_consensus.cc:3037] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:22.103868 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b0f95809434c3e898070fcaf3882ae" candidate_uuid: "3d7fc86bf79447ff8d354d47e6eded95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068558fbb8f4638ac9bfbff3265411f"
I20260619 06:48:22.103988 16215 raft_consensus.cc:3037] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:22.104326 16073 raft_consensus.cc:2454] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7fc86bf79447ff8d354d47e6eded95 in term 1.
I20260619 06:48:22.104606 16286 leader_election.cc:302] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7fc86bf79447ff8d354d47e6eded95, 435effe2c7ee434cb8117bb10f44e029; no voters:
I20260619 06:48:22.104768 16431 raft_consensus.cc:2781] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:22.104950 16431 raft_consensus.cc:686] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 LEADER]: Becoming Leader. State: Replica: 3d7fc86bf79447ff8d354d47e6eded95, State: Running, Role: LEADER
I20260619 06:48:22.105268 16431 consensus_queue.cc:237] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.105592 16215 raft_consensus.cc:2454] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7fc86bf79447ff8d354d47e6eded95 in term 1.
I20260619 06:48:22.108244 15918 catalog_manager.cc:5495] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d7fc86bf79447ff8d354d47e6eded95 (127.14.116.195). New cstate: current_term: 1 leader_uuid: "3d7fc86bf79447ff8d354d47e6eded95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } health_report { overall_health: HEALTHY } } }
I20260619 06:48:22.136191 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.132536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.14.116.195:35657)
0619 06:48:22.132679 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.132684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.132697 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:48:22.132903 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.132911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.132963 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.133075 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:22.133347 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.133845 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.134736 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.134967 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.135078 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.135395 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:22.135798 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:22.135917 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.136005 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.136054 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:48:22.141423 16215 raft_consensus.cc:1261] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [term 1 FOLLOWER]: Refusing update from remote peer 3d7fc86bf79447ff8d354d47e6eded95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:22.141433 16073 raft_consensus.cc:1261] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [term 1 FOLLOWER]: Refusing update from remote peer 3d7fc86bf79447ff8d354d47e6eded95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:22.142146 16434 consensus_queue.cc:1035] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:22.142313 16431 consensus_queue.cc:1035] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:48:22.146375 16402 tablet.cc:2369] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:22.150117 16439 mvcc.cc:205] Tried to move back new op lower bound from 7298464571964661760 to 7298464571825995776. Current Snapshot: MvccSnapshot[applied={T|T < 7298464571964661760}]
I20260619 06:48:22.151087 16438 mvcc.cc:205] Tried to move back new op lower bound from 7298464571964661760 to 7298464571825995776. Current Snapshot: MvccSnapshot[applied={T|T < 7298464571964661760}]
I20260619 06:48:22.160511 15916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:22.160655 15916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:22.162839 15916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307: Sending DeleteTablet for 3 replicas of tablet c6b0f95809434c3e898070fcaf3882ae
I20260619 06:48:22.163584 16195 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b0f95809434c3e898070fcaf3882ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:22 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:59368
I20260619 06:48:22.163597 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b0f95809434c3e898070fcaf3882ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:22 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:37858
I20260619 06:48:22.163674 16335 tablet_service.cc:1514] Processing DeleteTablet for tablet c6b0f95809434c3e898070fcaf3882ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:22 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48288
I20260619 06:48:22.164088 16448 tablet_replica.cc:330] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: stopping tablet replica
I20260619 06:48:22.164337 16447 tablet_replica.cc:330] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: stopping tablet replica
I20260619 06:48:22.164395 16448 raft_consensus.cc:2229] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:22.164603 16447 raft_consensus.cc:2229] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:22.164769 16448 raft_consensus.cc:2258] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:22.164848 16447 raft_consensus.cc:2258] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:22.164937 15916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:48:22.165119 16449 tablet_replica.cc:330] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: stopping tablet replica
W20260619 06:48:22.165318 15916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:22.165338 16449 raft_consensus.cc:2229] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:22.165532 16449 raft_consensus.cc:2258] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:22.167219 16448 ts_tablet_manager.cc:1905] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:22.167219 16447 ts_tablet_manager.cc:1905] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:22.168797 16449 ts_tablet_manager.cc:1905] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:22.169981 16447 ts_tablet_manager.cc:1918] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:22.170128 16447 log.cc:1196] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/c6b0f95809434c3e898070fcaf3882ae
I20260619 06:48:22.170461 16447 ts_tablet_manager.cc:1939] T c6b0f95809434c3e898070fcaf3882ae P a068558fbb8f4638ac9bfbff3265411f: Deleting consensus metadata
I20260619 06:48:22.170661 16335 tablet_service.cc:1467] Processing CreateTablet for tablet 9c590aa9690d459bb8a3c0772e25f32e (DEFAULT_TABLE table=test-table [id=6d8264c7135e463794e6ec393b20895d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:22.171262 16055 tablet_service.cc:1467] Processing CreateTablet for tablet 9c590aa9690d459bb8a3c0772e25f32e (DEFAULT_TABLE table=test-table [id=6d8264c7135e463794e6ec393b20895d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:22.171518 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c590aa9690d459bb8a3c0772e25f32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:22.171551 16335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c590aa9690d459bb8a3c0772e25f32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:22.171892 15904 catalog_manager.cc:4841] TS a068558fbb8f4638ac9bfbff3265411f (127.14.116.194:42085): tablet c6b0f95809434c3e898070fcaf3882ae (table test-table [id=c81aec797a4f43eda9423c524104cbcf]) successfully deleted
I20260619 06:48:22.172763 16195 tablet_service.cc:1467] Processing CreateTablet for tablet 9c590aa9690d459bb8a3c0772e25f32e (DEFAULT_TABLE table=test-table [id=6d8264c7135e463794e6ec393b20895d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:22.172988 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c590aa9690d459bb8a3c0772e25f32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:22.173647 16426 tablet_bootstrap.cc:492] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: Bootstrap starting.
I20260619 06:48:22.174600 16426 tablet_bootstrap.cc:654] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:22.174749 16448 ts_tablet_manager.cc:1918] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:22.174829 16448 log.cc:1196] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/c6b0f95809434c3e898070fcaf3882ae
I20260619 06:48:22.175045 16424 tablet_bootstrap.cc:492] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: Bootstrap starting.
I20260619 06:48:22.175141 16448 ts_tablet_manager.cc:1939] T c6b0f95809434c3e898070fcaf3882ae P 3d7fc86bf79447ff8d354d47e6eded95: Deleting consensus metadata
I20260619 06:48:22.175889 16424 tablet_bootstrap.cc:654] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:22.176471 15905 catalog_manager.cc:4841] TS 3d7fc86bf79447ff8d354d47e6eded95 (127.14.116.195:35657): tablet c6b0f95809434c3e898070fcaf3882ae (table test-table [id=c81aec797a4f43eda9423c524104cbcf]) successfully deleted
I20260619 06:48:22.177290 16449 ts_tablet_manager.cc:1918] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:22.177392 16449 log.cc:1196] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/c6b0f95809434c3e898070fcaf3882ae
I20260619 06:48:22.177313 16425 tablet_bootstrap.cc:492] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: Bootstrap starting.
I20260619 06:48:22.177713 16449 ts_tablet_manager.cc:1939] T c6b0f95809434c3e898070fcaf3882ae P 435effe2c7ee434cb8117bb10f44e029: Deleting consensus metadata
I20260619 06:48:22.178187 16426 tablet_bootstrap.cc:492] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: No bootstrap required, opened a new log
I20260619 06:48:22.178263 16426 ts_tablet_manager.cc:1397] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 06:48:22.178385 16425 tablet_bootstrap.cc:654] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:22.178669 16426 raft_consensus.cc:348] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.178774 16426 raft_consensus.cc:374] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:22.178817 16426 raft_consensus.cc:729] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7fc86bf79447ff8d354d47e6eded95, State: Initialized, Role: FOLLOWER
I20260619 06:48:22.178910 15902 catalog_manager.cc:4841] TS 435effe2c7ee434cb8117bb10f44e029 (127.14.116.193:46361): tablet c6b0f95809434c3e898070fcaf3882ae (table test-table [id=c81aec797a4f43eda9423c524104cbcf]) successfully deleted
I20260619 06:48:22.178941 16426 consensus_queue.cc:260] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.179222 16426 ts_tablet_manager.cc:1428] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:22.179940 16424 tablet_bootstrap.cc:492] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: No bootstrap required, opened a new log
I20260619 06:48:22.180016 16424 ts_tablet_manager.cc:1397] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 06:48:22.180405 16424 raft_consensus.cc:348] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.180514 16424 raft_consensus.cc:374] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:22.180544 16424 raft_consensus.cc:729] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068558fbb8f4638ac9bfbff3265411f, State: Initialized, Role: FOLLOWER
I20260619 06:48:22.180651 16424 consensus_queue.cc:260] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.180869 16424 ts_tablet_manager.cc:1428] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:22.181054 16425 tablet_bootstrap.cc:492] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: No bootstrap required, opened a new log
I20260619 06:48:22.181129 16425 ts_tablet_manager.cc:1397] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:48:22.181552 16425 raft_consensus.cc:348] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.181689 16425 raft_consensus.cc:374] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:22.181732 16425 raft_consensus.cc:729] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435effe2c7ee434cb8117bb10f44e029, State: Initialized, Role: FOLLOWER
I20260619 06:48:22.181841 16425 consensus_queue.cc:260] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.182126 16425 ts_tablet_manager.cc:1428] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260619 06:48:22.191835 16262 tablet.cc:2369] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:22.199990 16430 raft_consensus.cc:482] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:22.200105 16430 raft_consensus.cc:504] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.201263 16430 leader_election.cc:288] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 435effe2c7ee434cb8117bb10f44e029 (127.14.116.193:46361), 3d7fc86bf79447ff8d354d47e6eded95 (127.14.116.195:35657)
I20260619 06:48:22.205195 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.201453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46361 (local address 127.14.116.194:40445)
0619 06:48:22.201730 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.201745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.201900 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.202270 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.202276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.202299 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:22.202423 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.202433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.203264 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.203266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.204127 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.204135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.204764 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.204767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.204861 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.204900 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.204977 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.205045 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":67,"threads_started":1}
I20260619 06:48:22.205206 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.201580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:40445 (local address 127.14.116.193:46361)
0619 06:48:22.201691 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.201696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.201822 (+ 126us) server_negotiation.cc:407] Connection header received
0619 06:48:22.202009 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.202012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.202063 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.202146 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:22.202554 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.203152 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.204247 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.204639 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.204764 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.204818 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.205010 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.205089 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:48:22.205667 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c590aa9690d459bb8a3c0772e25f32e" candidate_uuid: "a068558fbb8f4638ac9bfbff3265411f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435effe2c7ee434cb8117bb10f44e029" is_pre_election: true
I20260619 06:48:22.205806 16073 raft_consensus.cc:2454] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a068558fbb8f4638ac9bfbff3265411f in term 0.
I20260619 06:48:22.206121 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.201469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:35657 (local address 127.14.116.194:58107)
0619 06:48:22.201938 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.201951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.202069 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.202390 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.202393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.202407 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:22.202516 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.202521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.203506 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.203509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.204752 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.204761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.205716 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.205720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.205825 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.205848 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.205934 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.205997 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":60,"threads_started":1}
I20260619 06:48:22.206115 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.201756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:58107 (local address 127.14.116.195:35657)
0619 06:48:22.201865 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.201869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.202009 (+ 140us) server_negotiation.cc:407] Connection header received
0619 06:48:22.202178 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.202181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.202222 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.202268 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:22.202629 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.203360 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.204917 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.205533 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.205634 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.205689 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.205948 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.206011 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:48:22.206147 16146 leader_election.cc:302] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435effe2c7ee434cb8117bb10f44e029, a068558fbb8f4638ac9bfbff3265411f; no voters:
I20260619 06:48:22.206393 16430 raft_consensus.cc:2781] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:22.206449 16430 raft_consensus.cc:482] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:22.206486 16430 raft_consensus.cc:3037] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:22.206631 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c590aa9690d459bb8a3c0772e25f32e" candidate_uuid: "a068558fbb8f4638ac9bfbff3265411f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7fc86bf79447ff8d354d47e6eded95" is_pre_election: true
I20260619 06:48:22.206887 16355 raft_consensus.cc:2454] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a068558fbb8f4638ac9bfbff3265411f in term 0.
I20260619 06:48:22.207654 16430 raft_consensus.cc:504] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.208000 16430 leader_election.cc:288] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [CANDIDATE]: Term 1 election: Requested vote from peers 435effe2c7ee434cb8117bb10f44e029 (127.14.116.193:46361), 3d7fc86bf79447ff8d354d47e6eded95 (127.14.116.195:35657)
I20260619 06:48:22.208300 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c590aa9690d459bb8a3c0772e25f32e" candidate_uuid: "a068558fbb8f4638ac9bfbff3265411f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435effe2c7ee434cb8117bb10f44e029"
I20260619 06:48:22.208297 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c590aa9690d459bb8a3c0772e25f32e" candidate_uuid: "a068558fbb8f4638ac9bfbff3265411f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7fc86bf79447ff8d354d47e6eded95"
I20260619 06:48:22.208436 16355 raft_consensus.cc:3037] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:22.208431 16073 raft_consensus.cc:3037] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:22.209591 16355 raft_consensus.cc:2454] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068558fbb8f4638ac9bfbff3265411f in term 1.
I20260619 06:48:22.209616 16073 raft_consensus.cc:2454] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068558fbb8f4638ac9bfbff3265411f in term 1.
I20260619 06:48:22.209851 16149 leader_election.cc:302] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7fc86bf79447ff8d354d47e6eded95, a068558fbb8f4638ac9bfbff3265411f; no voters:
I20260619 06:48:22.210057 16430 raft_consensus.cc:2781] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:22.210237 16430 raft_consensus.cc:686] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 1 LEADER]: Becoming Leader. State: Replica: a068558fbb8f4638ac9bfbff3265411f, State: Running, Role: LEADER
I20260619 06:48:22.210482 16430 consensus_queue.cc:237] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.212638 15918 catalog_manager.cc:5495] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f reported cstate change: term changed from 0 to 1, leader changed from <none> to a068558fbb8f4638ac9bfbff3265411f (127.14.116.194). New cstate: current_term: 1 leader_uuid: "a068558fbb8f4638ac9bfbff3265411f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } health_report { overall_health: UNKNOWN } } }
Jun 19 06:48:22 dist-test-slave-9x3b krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:22.245730 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.237212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44269 (local address 127.0.0.1:48878)
0619 06:48:22.237514 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.237527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.237633 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.237886 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.237888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.238236 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:22.238323 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.238329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.239335 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.239337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.239928 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.239936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.240468 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.240471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.240528 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.241073 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:22.241084 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:22.242532 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:22.244336 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:22.244343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:22.244348 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:22.244556 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:22.244815 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:22.244816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:22.244818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:22.244895 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:22.245261 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:22.245265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:22.245354 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.245439 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.245610 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20260619 06:48:22.245849 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.237273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.14.116.254:44269)
0619 06:48:22.237376 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.237380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.237612 (+ 232us) server_negotiation.cc:407] Connection header received
0619 06:48:22.237716 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.237719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.237766 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.237841 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:22.238435 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.239230 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.240025 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.240358 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.240466 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.242655 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:22.242659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:22.242662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:22.242686 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:22.244241 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:22.244645 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:22.244648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:22.244650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:22.244731 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:22.244975 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:22.244978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:22.244980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:22.245169 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:22.245261 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.245516 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.245676 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":38}
I20260619 06:48:22.249706 15918 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48878:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:48:22.250439 15918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:22.254941 16055 tablet_service.cc:1467] Processing CreateTablet for tablet 5e14c7e1bc814c01964f39225328afdd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=309e461f62a749f5bde811b74f0f0c8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:48:22.255043 16195 tablet_service.cc:1467] Processing CreateTablet for tablet 5e14c7e1bc814c01964f39225328afdd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=309e461f62a749f5bde811b74f0f0c8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:48:22.255113 16335 tablet_service.cc:1467] Processing CreateTablet for tablet 5e14c7e1bc814c01964f39225328afdd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=309e461f62a749f5bde811b74f0f0c8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:48:22.255288 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e14c7e1bc814c01964f39225328afdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:22.255393 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e14c7e1bc814c01964f39225328afdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:22.255414 16335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e14c7e1bc814c01964f39225328afdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:22.257965 16426 tablet_bootstrap.cc:492] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: Bootstrap starting.
I20260619 06:48:22.258234 16425 tablet_bootstrap.cc:492] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: Bootstrap starting.
I20260619 06:48:22.258791 16426 tablet_bootstrap.cc:654] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:22.259183 16425 tablet_bootstrap.cc:654] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:22.260347 16426 tablet_bootstrap.cc:492] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: No bootstrap required, opened a new log
I20260619 06:48:22.260443 16426 ts_tablet_manager.cc:1397] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:48:22.260789 16426 raft_consensus.cc:348] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.260905 16426 raft_consensus.cc:374] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:22.260943 16426 raft_consensus.cc:729] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7fc86bf79447ff8d354d47e6eded95, State: Initialized, Role: FOLLOWER
I20260619 06:48:22.261082 16426 consensus_queue.cc:260] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.261336 16442 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.261477 16442 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:22.261538 16424 tablet_bootstrap.cc:492] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: Bootstrap starting.
I20260619 06:48:22.261353 16426 ts_tablet_manager.cc:1428] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:48:22.262488 16424 tablet_bootstrap.cc:654] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:22.263747 16424 tablet_bootstrap.cc:492] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: No bootstrap required, opened a new log
I20260619 06:48:22.263825 16424 ts_tablet_manager.cc:1397] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:22.264308 16425 tablet_bootstrap.cc:492] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: No bootstrap required, opened a new log
I20260619 06:48:22.264284 16424 raft_consensus.cc:348] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.264387 16425 ts_tablet_manager.cc:1397] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 06:48:22.264391 16424 raft_consensus.cc:374] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:22.264429 16424 raft_consensus.cc:729] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068558fbb8f4638ac9bfbff3265411f, State: Initialized, Role: FOLLOWER
I20260619 06:48:22.264549 16424 consensus_queue.cc:260] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.264794 16424 ts_tablet_manager.cc:1428] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:22.264780 16425 raft_consensus.cc:348] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.264887 16425 raft_consensus.cc:374] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:22.264930 16425 raft_consensus.cc:729] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435effe2c7ee434cb8117bb10f44e029, State: Initialized, Role: FOLLOWER
I20260619 06:48:22.265034 16430 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.265065 16425 consensus_queue.cc:260] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.265141 16430 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:22.265305 16425 ts_tablet_manager.cc:1428] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:22.265334 16432 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.265470 16432 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:22.644060 16432 raft_consensus.cc:482] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:22.644192 16432 raft_consensus.cc:504] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.645294 16432 leader_election.cc:288] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a068558fbb8f4638ac9bfbff3265411f (127.14.116.194:42085), 3d7fc86bf79447ff8d354d47e6eded95 (127.14.116.195:35657)
I20260619 06:48:22.650245 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.645703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:48421 (local address 127.14.116.194:42085)
0619 06:48:22.645906 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.645912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.646074 (+ 162us) server_negotiation.cc:407] Connection header received
0619 06:48:22.646216 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.646219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.646259 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.646364 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:22.646800 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.647543 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.649180 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.649698 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.649815 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.649870 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.650039 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.650114 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":54,"threads_started":1}
I20260619 06:48:22.650262 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.645692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:32883 (local address 127.14.116.195:35657)
0619 06:48:22.645827 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.645843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.646020 (+ 177us) server_negotiation.cc:407] Connection header received
0619 06:48:22.646167 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.646173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.646230 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.646287 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:22.646601 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.647357 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.649125 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.649722 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.649862 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.649909 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.650082 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.650144 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:48:22.650290 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.645546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:35657 (local address 127.14.116.193:32883)
0619 06:48:22.645905 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.645921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.646034 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.646321 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.646326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.646347 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:22.646498 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.646503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.647467 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.647471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.649014 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.649023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.649858 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.649862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.649964 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.649990 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.650056 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.650129 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":23,"thread_start_us":118,"threads_started":1}
I20260619 06:48:22.650244 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.645569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:42085 (local address 127.14.116.193:48421)
0619 06:48:22.645956 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.645967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.646114 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.646394 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.646396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.646411 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:22.646518 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.646522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.647708 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.647712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.649076 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.649087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.649812 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.649817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.649913 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.649943 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.650018 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.650082 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":18,"thread_start_us":104,"threads_started":1}
I20260619 06:48:22.650825 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e14c7e1bc814c01964f39225328afdd" candidate_uuid: "435effe2c7ee434cb8117bb10f44e029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068558fbb8f4638ac9bfbff3265411f" is_pre_election: true
I20260619 06:48:22.650882 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e14c7e1bc814c01964f39225328afdd" candidate_uuid: "435effe2c7ee434cb8117bb10f44e029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7fc86bf79447ff8d354d47e6eded95" is_pre_election: true
I20260619 06:48:22.650981 16215 raft_consensus.cc:2454] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435effe2c7ee434cb8117bb10f44e029 in term 0.
I20260619 06:48:22.650992 16355 raft_consensus.cc:2454] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435effe2c7ee434cb8117bb10f44e029 in term 0.
I20260619 06:48:22.651309 16008 leader_election.cc:302] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435effe2c7ee434cb8117bb10f44e029, a068558fbb8f4638ac9bfbff3265411f; no voters:
I20260619 06:48:22.651582 16432 raft_consensus.cc:2781] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:22.651664 16432 raft_consensus.cc:482] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:22.651710 16432 raft_consensus.cc:3037] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:22.652817 16432 raft_consensus.cc:504] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.653132 16432 leader_election.cc:288] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [CANDIDATE]: Term 1 election: Requested vote from peers a068558fbb8f4638ac9bfbff3265411f (127.14.116.194:42085), 3d7fc86bf79447ff8d354d47e6eded95 (127.14.116.195:35657)
I20260619 06:48:22.653406 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e14c7e1bc814c01964f39225328afdd" candidate_uuid: "435effe2c7ee434cb8117bb10f44e029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068558fbb8f4638ac9bfbff3265411f"
I20260619 06:48:22.653489 16355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e14c7e1bc814c01964f39225328afdd" candidate_uuid: "435effe2c7ee434cb8117bb10f44e029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7fc86bf79447ff8d354d47e6eded95"
I20260619 06:48:22.653518 16215 raft_consensus.cc:3037] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:22.653594 16355 raft_consensus.cc:3037] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:22.654621 16215 raft_consensus.cc:2454] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435effe2c7ee434cb8117bb10f44e029 in term 1.
I20260619 06:48:22.654644 16355 raft_consensus.cc:2454] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435effe2c7ee434cb8117bb10f44e029 in term 1.
I20260619 06:48:22.654904 16009 leader_election.cc:302] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7fc86bf79447ff8d354d47e6eded95, 435effe2c7ee434cb8117bb10f44e029; no voters:
I20260619 06:48:22.655081 16432 raft_consensus.cc:2781] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:22.655251 16432 raft_consensus.cc:686] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [term 1 LEADER]: Becoming Leader. State: Replica: 435effe2c7ee434cb8117bb10f44e029, State: Running, Role: LEADER
I20260619 06:48:22.655491 16432 consensus_queue.cc:237] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } }
I20260619 06:48:22.656777 16469 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: TxnStatusTablet state changed. Reason: New leader 435effe2c7ee434cb8117bb10f44e029. Latest consensus state: current_term: 1 leader_uuid: "435effe2c7ee434cb8117bb10f44e029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.656883 16469 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:48:22.657146 16471 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:48:22.657305 16471 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:48:22.657680 15916 catalog_manager.cc:5495] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 reported cstate change: term changed from 0 to 1, leader changed from <none> to 435effe2c7ee434cb8117bb10f44e029 (127.14.116.193). New cstate: current_term: 1 leader_uuid: "435effe2c7ee434cb8117bb10f44e029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } health_report { overall_health: UNKNOWN } } }
I20260619 06:48:22.716261 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.712164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.14.116.193:46361)
0619 06:48:22.712383 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.712387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.712401 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:22.712529 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.712531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.712583 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.712667 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:22.713008 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.713581 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.714784 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.715023 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.715106 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.715418 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:22.715884 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:22.715967 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.716092 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.716139 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
I20260619 06:48:22.716280 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.712026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46361 (local address 127.0.0.1:37860)
0619 06:48:22.712255 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.712270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.712377 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.712775 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.712778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.712787 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:22.712898 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.712904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.713680 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.713683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.714661 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.714671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.715133 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.715137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.715225 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.715275 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:22.715977 (+ 702us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:22.715991 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.716064 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.716146 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260619 06:48:22.719116 16215 raft_consensus.cc:1261] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f [term 1 FOLLOWER]: Refusing update from remote peer 435effe2c7ee434cb8117bb10f44e029: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:22.719285 16355 raft_consensus.cc:1261] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 FOLLOWER]: Refusing update from remote peer 435effe2c7ee434cb8117bb10f44e029: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:22.719751 16432 consensus_queue.cc:1035] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:22.720189 16432 consensus_queue.cc:1035] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:22.721395 16430 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: TxnStatusTablet state changed. Reason: New leader 435effe2c7ee434cb8117bb10f44e029. Latest consensus state: current_term: 1 leader_uuid: "435effe2c7ee434cb8117bb10f44e029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.721527 16430 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:22.722102 16442 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: TxnStatusTablet state changed. Reason: New leader 435effe2c7ee434cb8117bb10f44e029. Latest consensus state: current_term: 1 leader_uuid: "435effe2c7ee434cb8117bb10f44e029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.722208 16442 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:22.723965 16432 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "435effe2c7ee434cb8117bb10f44e029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.724067 16432 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:48:22.724169 16480 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "435effe2c7ee434cb8117bb10f44e029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.724340 16480 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P 435effe2c7ee434cb8117bb10f44e029: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:48:22.724363 16442 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "435effe2c7ee434cb8117bb10f44e029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.724463 16442 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P 3d7fc86bf79447ff8d354d47e6eded95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:22.725013 16476 mvcc.cc:205] Tried to move back new op lower bound from 7298464574332604416 to 7298464574079225856. Current Snapshot: MvccSnapshot[applied={T|T < 7298464574332604416}]
I20260619 06:48:22.725750 16430 tablet_replica.cc:439] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "435effe2c7ee434cb8117bb10f44e029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 } } peers { permanent_uuid: "a068558fbb8f4638ac9bfbff3265411f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42085 } } peers { permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 } } }
I20260619 06:48:22.725854 16430 tablet_replica.cc:442] T 5e14c7e1bc814c01964f39225328afdd P a068558fbb8f4638ac9bfbff3265411f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:22.726850 16430 consensus_queue.cc:1035] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7fc86bf79447ff8d354d47e6eded95" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:22.731119 16485 consensus_queue.cc:1035] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [LEADER]: Connected to new peer: Peer: permanent_uuid: "435effe2c7ee434cb8117bb10f44e029" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:22.739892 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.736250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.14.116.194:42085)
0619 06:48:22.736356 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.736360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.736588 (+ 228us) server_negotiation.cc:407] Connection header received
0619 06:48:22.736765 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.736768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.736812 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.736911 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:22.737170 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.737671 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.738506 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.738723 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.738827 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.739113 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:22.739507 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:22.739647 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.739712 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.739765 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:48:22.749756 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.746197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:60309 (local address 127.14.116.193:46361)
0619 06:48:22.746301 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.746306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.746539 (+ 233us) server_negotiation.cc:407] Connection header received
0619 06:48:22.746659 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.746662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.746713 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.746790 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:22.747073 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.747848 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.748690 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.749031 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.749130 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.749371 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:22.749443 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:22.749512 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.749586 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.749633 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:48:22.749786 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.746114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46361 (local address 127.14.116.194:60309)
0619 06:48:22.746450 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.746463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.746545 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.746814 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.746816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.746822 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:22.746896 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.746899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.747942 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.747945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.748584 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.748593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.749132 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.749134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.749205 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.749243 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:22.749521 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:22.749526 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.749576 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.749632 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":87,"threads_started":1}
I20260619 06:48:22.773538 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.769896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:41629 (local address 127.14.116.194:42085)
0619 06:48:22.770014 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.770018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.770227 (+ 209us) server_negotiation.cc:407] Connection header received
0619 06:48:22.770366 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.770368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.770404 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.770462 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:22.770765 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.771383 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.772597 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.772812 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.772889 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.773179 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:22.773238 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:22.773294 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.773402 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.773440 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:22.773613 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.769847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:42085 (local address 127.14.116.193:41629)
0619 06:48:22.770157 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:22.770172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:22.770279 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:22.770495 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:22.770498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:22.770506 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:22.770627 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.770634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.771491 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.771495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.772483 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:22.772494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.772919 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.772923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:22.773016 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.773068 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:22.773317 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:22.773323 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:48:22.773395 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 06:48:22.773476 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":60,"threads_started":1}
I20260619 06:48:22.793292 15916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:22.793470 15916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:22.795413 15916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ff6262ce08a440db109aaa995d9a307: Sending DeleteTablet for 3 replicas of tablet 9c590aa9690d459bb8a3c0772e25f32e
I20260619 06:48:22.795966 16195 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c590aa9690d459bb8a3c0772e25f32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:22 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:59368
I20260619 06:48:22.796069 16335 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c590aa9690d459bb8a3c0772e25f32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:22 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48288
I20260619 06:48:22.796396 16499 tablet_replica.cc:330] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: stopping tablet replica
I20260619 06:48:22.796515 16499 raft_consensus.cc:2229] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:22.796622 16499 raft_consensus.cc:2258] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:22.797317 16499 ts_tablet_manager.cc:1905] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:22.797684 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c590aa9690d459bb8a3c0772e25f32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:22 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:37858
I20260619 06:48:22.798050 16498 tablet_replica.cc:330] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: stopping tablet replica
I20260619 06:48:22.798187 16498 raft_consensus.cc:2229] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:22.798461 16498 raft_consensus.cc:2258] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:22.799187 16498 ts_tablet_manager.cc:1905] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:22.799788 16501 tablet_replica.cc:330] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: stopping tablet replica
I20260619 06:48:22.799993 16501 raft_consensus.cc:2229] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:22.800136 16501 raft_consensus.cc:2258] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:22.800670 16499 ts_tablet_manager.cc:1918] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:22.800752 16499 log.cc:1196] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/9c590aa9690d459bb8a3c0772e25f32e
I20260619 06:48:22.801023 16499 ts_tablet_manager.cc:1939] T 9c590aa9690d459bb8a3c0772e25f32e P 3d7fc86bf79447ff8d354d47e6eded95: Deleting consensus metadata
I20260619 06:48:22.801420 16501 ts_tablet_manager.cc:1905] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:22.801636 15905 catalog_manager.cc:4841] TS 3d7fc86bf79447ff8d354d47e6eded95 (127.14.116.195:35657): tablet 9c590aa9690d459bb8a3c0772e25f32e (table test-table [id=6d8264c7135e463794e6ec393b20895d]) successfully deleted
I20260619 06:48:22.803472 16498 ts_tablet_manager.cc:1918] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:22.803562 16498 log.cc:1196] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/9c590aa9690d459bb8a3c0772e25f32e
I20260619 06:48:22.803848 16498 ts_tablet_manager.cc:1939] T 9c590aa9690d459bb8a3c0772e25f32e P a068558fbb8f4638ac9bfbff3265411f: Deleting consensus metadata
I20260619 06:48:22.804441 15904 catalog_manager.cc:4841] TS a068558fbb8f4638ac9bfbff3265411f (127.14.116.194:42085): tablet 9c590aa9690d459bb8a3c0772e25f32e (table test-table [id=6d8264c7135e463794e6ec393b20895d]) successfully deleted
I20260619 06:48:22.805943 16501 ts_tablet_manager.cc:1918] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:22.806039 16501 log.cc:1196] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/9c590aa9690d459bb8a3c0772e25f32e
I20260619 06:48:22.806283 16501 ts_tablet_manager.cc:1939] T 9c590aa9690d459bb8a3c0772e25f32e P 435effe2c7ee434cb8117bb10f44e029: Deleting consensus metadata
I20260619 06:48:22.806885 15902 catalog_manager.cc:4841] TS 435effe2c7ee434cb8117bb10f44e029 (127.14.116.193:46361): tablet 9c590aa9690d459bb8a3c0772e25f32e (table test-table [id=6d8264c7135e463794e6ec393b20895d]) successfully deleted
Jun 19 06:48:22 dist-test-slave-9x3b krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851701, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:22.816015 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.799614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.14.116.193:46361)
0619 06:48:22.799789 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.799794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.801770 (+ 1976us) server_negotiation.cc:407] Connection header received
0619 06:48:22.806959 (+ 5189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.806962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.807014 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.807069 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:22.807850 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.808350 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.809148 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.809384 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.809491 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.812507 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:22.812524 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:22.812535 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:22.812556 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:22.814286 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:22.814736 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:22.814743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:22.814745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:22.814824 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:22.815094 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:22.815097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:22.815099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:22.815341 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:22.815601 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.815724 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.815839 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 06:48:22.816502 16105 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37868
I20260619 06:48:22.825563 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:22.818173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.14.116.254:44269)
0619 06:48:22.818394 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:22.818399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:22.818412 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:48:22.818505 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:22.818508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:22.818552 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:22.818649 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:22.819361 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.820017 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.820808 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:22.821111 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:22.821280 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:22.822489 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:22.822493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:22.822495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:22.822518 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:22.824026 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:22.824411 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:22.824416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:22.824417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:22.824496 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:22.824753 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:22.824756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:22.824757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:22.824921 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:22.825052 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:22.825266 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 06:48:22.825377 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":52,"threads_started":1}
W20260619 06:48:22.825874 15916 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48884
I20260619 06:48:22.827225 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 15990
I20260619 06:48:22.835253 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 16130
I20260619 06:48:22.841178 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 16270
I20260619 06:48:22.847313 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 15886
2026-06-19T06:48:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2955 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:22 dist-test-slave-9x3b krb5kdc[16508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:22 dist-test-slave-9x3b krb5kdc[16508](info): set up 2 sockets
Jun 19 06:48:22 dist-test-slave-9x3b krb5kdc[16508](info): commencing operation
krb5kdc: starting...
W20260619 06:48:23.875257 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:23 dist-test-slave-9x3b krb5kdc[16508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:23.988440 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:41549
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:41549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:24.092551 16521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:24.092785 16521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:24.092823 16521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:24.096696 16521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:24.096745 16521 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:48:24.096762 16521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:24.096781 16521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:24.096796 16521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:24.096810 16521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:24.101838 16521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:41549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:24.102701 16521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:24.103703 16521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:24.112418 16526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:24.112421 16529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:24.112674 16521 server_base.cc:1034] running on GCE node
W20260619 06:48:24.112748 16527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:24.113255 16521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:24.114173 16521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:24.115334 16521 hybrid_clock.cc:648] HybridClock initialized: now 1781851704115301 us; error 45 us; skew 500 ppm
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:24.117952 16521 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:24.118772 16521 webserver.cc:460] Webserver started at http://127.14.116.254:40819/ using document root <none> and password file <none>
I20260619 06:48:24.119288 16521 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:24.119375 16521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:24.119596 16521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:24.121501 16521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "02773ce996d2496189c5a945bd1e1b33"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "c3319ed17b54790c2dec810a496856a1"
server_key_iv: "847d153e63145140036094a3c18c0f52"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.122047 16521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "02773ce996d2496189c5a945bd1e1b33"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "c3319ed17b54790c2dec810a496856a1"
server_key_iv: "847d153e63145140036094a3c18c0f52"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.125545 16521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 06:48:24.127822 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:24.128963 16521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 06:48:24.129102 16521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "02773ce996d2496189c5a945bd1e1b33"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "c3319ed17b54790c2dec810a496856a1"
server_key_iv: "847d153e63145140036094a3c18c0f52"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.129211 16521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:24.162097 16521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:24.165337 16521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:24.165550 16521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:24.173075 16521 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:41549
I20260619 06:48:24.173102 16598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:41549 every 8 connection(s)
I20260619 06:48:24.173938 16521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:24.176478 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:24.181414 16599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33: Bootstrap starting.
I20260619 06:48:24.183684 16599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:24.183885 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 16521
I20260619 06:48:24.184031 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:24.184294 14803 external_mini_cluster.cc:1428] Setting key e91bb4fb517e532607c6ab2063427c8b
I20260619 06:48:24.184551 16599 log.cc:824] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:24.186439 16599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33: No bootstrap required, opened a new log
I20260619 06:48:24.189646 16599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02773ce996d2496189c5a945bd1e1b33" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41549 } }
I20260619 06:48:24.189901 16599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:24.189963 16599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02773ce996d2496189c5a945bd1e1b33, State: Initialized, Role: FOLLOWER
I20260619 06:48:24.190407 16599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02773ce996d2496189c5a945bd1e1b33" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41549 } }
I20260619 06:48:24.190533 16599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:24.190618 16599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:24.190734 16599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:24.191810 16599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02773ce996d2496189c5a945bd1e1b33" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41549 } }
I20260619 06:48:24.192183 16599 leader_election.cc:302] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02773ce996d2496189c5a945bd1e1b33; no voters:
I20260619 06:48:24.192504 16599 leader_election.cc:288] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:24.192597 16604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:24.192826 16604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [term 1 LEADER]: Becoming Leader. State: Replica: 02773ce996d2496189c5a945bd1e1b33, State: Running, Role: LEADER
I20260619 06:48:24.193125 16604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02773ce996d2496189c5a945bd1e1b33" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41549 } }
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:24.193676 16599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:24.194416 16606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02773ce996d2496189c5a945bd1e1b33. Latest consensus state: current_term: 1 leader_uuid: "02773ce996d2496189c5a945bd1e1b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02773ce996d2496189c5a945bd1e1b33" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41549 } } }
I20260619 06:48:24.194872 16605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02773ce996d2496189c5a945bd1e1b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02773ce996d2496189c5a945bd1e1b33" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41549 } } }
I20260619 06:48:24.195042 16605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:24.195434 16613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:24.195645 16606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:24.198436 16613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:24.199707 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.185369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.14.116.254:41549)
0619 06:48:24.185767 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.185780 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.185812 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:48:24.186407 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.186423 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.186766 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.187024 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:24.187902 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.189210 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.190344 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.190978 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.191090 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.193956 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:24.193962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:24.193976 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:24.194005 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:24.196392 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.196844 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.196853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.196862 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.197008 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.197328 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.197333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.197336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.197807 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:24.197974 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.198634 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.198919 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260619 06:48:24.204506 16613 catalog_manager.cc:1347] Generated new cluster ID: 53147fabf8d54f23b18588e335d19cdd
I20260619 06:48:24.204566 16613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:24.215128 16613 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:24.216290 16613 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:24.233139 16613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33: Generated new TSK 0
I20260619 06:48:24.233860 16613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:24.268311 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:41549
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:24.384599 16625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:24.384838 16625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:24.384886 16625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:24.388819 16625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:24.388868 16625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:24.388906 16625 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:48:24.388924 16625 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:48:24.388955 16625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:24.394028 16625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.254:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:24.395076 16625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:24.396143 16625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:24.405033 16630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:24.405084 16631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:24.405033 16633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:24.405246 16625 server_base.cc:1034] running on GCE node
I20260619 06:48:24.405634 16625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:24.406210 16625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:24.407366 16625 hybrid_clock.cc:648] HybridClock initialized: now 1781851704407354 us; error 25 us; skew 500 ppm
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:24.410187 16625 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:24.410993 16625 webserver.cc:460] Webserver started at http://127.14.116.193:41569/ using document root <none> and password file <none>
I20260619 06:48:24.411526 16625 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:24.411583 16625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:24.411832 16625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:24.413805 16625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "1361158ef2a33a3be60ba765d14427f8"
server_key_iv: "6951b44808c3e69b52c36480b2d0d3c7"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.414356 16625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "1361158ef2a33a3be60ba765d14427f8"
server_key_iv: "6951b44808c3e69b52c36480b2d0d3c7"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.417974 16625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 06:48:24.420236 16640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:24.421375 16625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:48:24.421489 16625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "1361158ef2a33a3be60ba765d14427f8"
server_key_iv: "6951b44808c3e69b52c36480b2d0d3c7"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.421595 16625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:24.441233 16625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:24.444025 16625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:24.444219 16625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:24.445554 16625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:24.445600 16625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:24.445662 16625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:24.445694 16625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:24.457458 16625 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:39291
I20260619 06:48:24.457760 16755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:39291 every 8 connection(s)
I20260619 06:48:24.458356 16625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:24.460318 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.446366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:33993 (local address 127.14.116.254:41549)
0619 06:48:24.446538 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.446545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.447299 (+ 754us) server_negotiation.cc:407] Connection header received
0619 06:48:24.448370 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.448375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.448438 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.448497 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:24.450200 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.451089 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.452667 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.453110 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.453227 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.456604 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:24.456608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:24.456611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:24.456637 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:24.458232 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.458780 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.458783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.458785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.458884 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.459298 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.459302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.459304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.459484 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:24.459559 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.460107 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.460203 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:48:24.461086 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.446713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:41549 (local address 127.14.116.193:33993)
0619 06:48:24.447135 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:24.447183 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:24.448115 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:24.448927 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:24.448937 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:24.449375 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:24.450025 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.450048 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.451248 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.451252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.452540 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.452549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.453296 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.453306 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.453536 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.454152 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:24.454171 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:24.456420 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:24.458348 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.458354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.458367 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.458676 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.458992 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.458995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.458997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.459166 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.459581 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:24.459585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:24.459893 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 06:48:24.460041 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 06:48:24.460296 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":124,"threads_started":1}
I20260619 06:48:24.462451 16756 heartbeater.cc:347] Connected to a master server at 127.14.116.254:41549
I20260619 06:48:24.462605 16756 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:24.463112 16756 heartbeater.cc:511] Master 127.14.116.254:41549 requested a full tablet report, sending...
I20260619 06:48:24.464453 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 16625
I20260619 06:48:24.464597 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:24.464607 16552 ts_manager.cc:194] Registered new tserver with Master: 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291)
I20260619 06:48:24.464896 14803 external_mini_cluster.cc:1428] Setting key 394b3fa4d8891011cc218d4ffb6e0dd2
I20260619 06:48:24.466121 16552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:33993
I20260619 06:48:24.471868 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.464346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:57751 (local address 127.14.116.254:41549)
0619 06:48:24.464450 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.464454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.464662 (+ 208us) server_negotiation.cc:407] Connection header received
0619 06:48:24.464839 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.464842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.464884 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.464929 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:24.465484 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.466215 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.467136 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.467354 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.467462 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.468718 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:24.468721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:24.468724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:24.468747 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:24.470195 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.470716 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.470718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.470719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.470784 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.471081 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.471083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.471084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.471229 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:24.471299 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.471670 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.471766 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:24.472018 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.464259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:41549 (local address 127.14.116.193:57751)
0619 06:48:24.464586 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:24.464599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:24.464723 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:24.465053 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:24.465056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:24.465247 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:24.465356 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.465362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.466369 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.466372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.467015 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.467023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.467465 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.467468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.467559 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.468099 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:24.468117 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:24.468564 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:24.470333 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.470336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.470339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.470619 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.470875 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.470879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.470883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.470990 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.471328 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:24.471334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:24.471451 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 06:48:24.471569 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 06:48:24.471749 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:24.491711 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:41549
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:24.597797 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:24.598073 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:24.598150 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:24.602103 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:24.602182 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:24.602253 16765 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:48:24.602295 16765 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:48:24.602357 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:24.607273 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.254:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:24.608268 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:24.609211 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:24.618237 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:24.618227 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:24.618219 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:24.618289 16765 server_base.cc:1034] running on GCE node
I20260619 06:48:24.618736 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:24.619325 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:24.620515 16765 hybrid_clock.cc:648] HybridClock initialized: now 1781851704620507 us; error 28 us; skew 500 ppm
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:24.623099 16765 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:24.624019 16765 webserver.cc:460] Webserver started at http://127.14.116.194:43347/ using document root <none> and password file <none>
I20260619 06:48:24.624610 16765 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:24.624696 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:24.624897 16765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:24.626844 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "cd795e0e60e94a67838e492f67e31ef9"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "9f3acff06d6ca5d85f7b44d82589c68e"
server_key_iv: "cd9fd555cf06662be0ffbf1154dc8a03"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.627415 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "cd795e0e60e94a67838e492f67e31ef9"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "9f3acff06d6ca5d85f7b44d82589c68e"
server_key_iv: "cd9fd555cf06662be0ffbf1154dc8a03"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.630875 16765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:48:24.633247 16780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:24.634490 16765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:24.634619 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "cd795e0e60e94a67838e492f67e31ef9"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "9f3acff06d6ca5d85f7b44d82589c68e"
server_key_iv: "cd9fd555cf06662be0ffbf1154dc8a03"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.634727 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:24.663160 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:24.666440 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:24.666639 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:24.668169 16765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:24.668241 16765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:24.668313 16765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:24.668371 16765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:24.679028 16765 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:46333
I20260619 06:48:24.679082 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:46333 every 8 connection(s)
I20260619 06:48:24.680061 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:24.681699 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.669706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:41549 (local address 127.14.116.194:44857)
0619 06:48:24.670152 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:24.670201 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:24.671118 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:24.671712 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:24.671719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:24.672059 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:24.672646 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.672659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.673436 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.673439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.674440 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.674449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.675009 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.675014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.675170 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.675752 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:24.675770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:24.677225 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:24.679093 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.679100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.679114 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.679387 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.679706 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.679709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.679711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.679845 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.680277 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:24.680281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:24.680547 (+ 266us) client_negotiation.cc:769] Sending connection context
0619 06:48:24.680732 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 06:48:24.681017 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":139,"threads_started":1}
I20260619 06:48:24.682235 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.668885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:44857 (local address 127.14.116.254:41549)
0619 06:48:24.669036 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.669043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.670318 (+ 1275us) server_negotiation.cc:407] Connection header received
0619 06:48:24.671331 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.671335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.671390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.671495 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:24.672788 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.673306 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.674562 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.674845 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.674968 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.677392 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:24.677396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:24.677398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:24.677424 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:24.678968 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.679490 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.679494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.679496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.679591 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.679941 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.679945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.679946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.680178 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:24.680280 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.682043 (+ 1763us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.682135 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:48:24.683087 16896 heartbeater.cc:347] Connected to a master server at 127.14.116.254:41549
I20260619 06:48:24.683211 16896 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:24.683732 16896 heartbeater.cc:511] Master 127.14.116.254:41549 requested a full tablet report, sending...
I20260619 06:48:24.684785 16553 ts_manager.cc:194] Registered new tserver with Master: cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333)
I20260619 06:48:24.685586 16553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:44857
I20260619 06:48:24.688333 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 16765
I20260619 06:48:24.688490 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:24.688858 14803 external_mini_cluster.cc:1428] Setting key b510e5da47468ff275516ef20fa3eca4
I20260619 06:48:24.692337 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.685237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:41549 (local address 127.14.116.194:48043)
0619 06:48:24.685574 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:24.685586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:24.685711 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:24.685961 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:24.685964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:24.686160 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:24.686267 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.686274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.686969 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.686972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.687734 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.687743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.688149 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.688154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.688247 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.688743 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:24.688758 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:24.689139 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:24.690837 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.690843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.690848 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.691115 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.691361 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.691367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.691372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.691470 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.691967 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:24.691969 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:24.692040 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 06:48:24.692121 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 06:48:24.692216 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":90,"threads_started":1}
I20260619 06:48:24.692512 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.685316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:48043 (local address 127.14.116.254:41549)
0619 06:48:24.685470 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.685475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.685653 (+ 178us) server_negotiation.cc:407] Connection header received
0619 06:48:24.685821 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.685824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.685872 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.685932 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:24.686379 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.686846 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.687825 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.688028 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.688117 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.689276 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:24.689281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:24.689284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:24.689307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:24.690732 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.691213 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.691215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.691216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.691280 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.691554 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.691556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.691557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.691863 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:24.691982 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.692199 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.692321 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:24.713759 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:41549
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:24.818995 16905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:24.819211 16905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:24.819305 16905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:24.823213 16905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:24.823269 16905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:24.823354 16905 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:48:24.823382 16905 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:48:24.823437 16905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:24.828593 16905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.254:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:24.829470 16905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:24.830396 16905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:24.839243 16913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:24.839242 16911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:24.839326 16910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:24.839601 16905 server_base.cc:1034] running on GCE node
I20260619 06:48:24.839962 16905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:24.840579 16905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:24.841769 16905 hybrid_clock.cc:648] HybridClock initialized: now 1781851704841748 us; error 37 us; skew 500 ppm
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:24.844408 16905 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:24.845217 16905 webserver.cc:460] Webserver started at http://127.14.116.195:46867/ using document root <none> and password file <none>
I20260619 06:48:24.845700 16905 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:24.845754 16905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:24.845901 16905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:24.847828 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "445ec28deed947fea13fae313cc3ba0e"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "6d82d3e23f68987f255ff0ea2eebe44c"
server_key_iv: "0db21fb4f4c72a404f8399909ba43cbf"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.848312 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "445ec28deed947fea13fae313cc3ba0e"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "6d82d3e23f68987f255ff0ea2eebe44c"
server_key_iv: "0db21fb4f4c72a404f8399909ba43cbf"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.851694 16905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:48:24.853818 16920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:24.854921 16905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:48:24.855037 16905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "445ec28deed947fea13fae313cc3ba0e"
format_stamp: "Formatted at 2026-06-19 06:48:24 on dist-test-slave-9x3b"
server_key: "6d82d3e23f68987f255ff0ea2eebe44c"
server_key_iv: "0db21fb4f4c72a404f8399909ba43cbf"
server_key_version: "encryptionkey@0"
I20260619 06:48:24.855135 16905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:24.866587 16905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:24.869263 16905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:24.869450 16905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:24.870833 16905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:24.870905 16905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:24.870970 16905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:24.871024 16905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:24.881649 16905 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:34567
I20260619 06:48:24.881729 17035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:34567 every 8 connection(s)
I20260619 06:48:24.882632 16905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:24.884423 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.871438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:53045 (local address 127.14.116.254:41549)
0619 06:48:24.871578 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.871582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.872456 (+ 874us) server_negotiation.cc:407] Connection header received
0619 06:48:24.873324 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.873331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.873388 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.873490 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:24.874695 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.875526 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.876968 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.877185 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.877296 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.880689 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:24.880692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:24.880694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:24.880717 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:24.882266 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.882820 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.882829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.882831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.882902 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.883312 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.883316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.883318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.883551 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:24.883705 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.884179 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.884307 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:48:24.885177 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.871884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:41549 (local address 127.14.116.195:53045)
0619 06:48:24.872294 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:24.872339 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:24.873130 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:24.873685 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:24.873694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:24.874072 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:24.874515 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.874528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.875695 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.875699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.876843 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.876852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.877327 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.877331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.877506 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.878096 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:24.878117 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:24.880523 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:24.882373 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.882379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.882391 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.882716 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.882995 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.882999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.883001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.883160 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.883696 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:24.883702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:24.883978 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 06:48:24.884127 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:48:24.884350 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":118,"threads_started":1}
I20260619 06:48:24.886317 17036 heartbeater.cc:347] Connected to a master server at 127.14.116.254:41549
I20260619 06:48:24.886447 17036 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:24.886965 17036 heartbeater.cc:511] Master 127.14.116.254:41549 requested a full tablet report, sending...
I20260619 06:48:24.887979 16552 ts_manager.cc:194] Registered new tserver with Master: 445ec28deed947fea13fae313cc3ba0e (127.14.116.195:34567)
I20260619 06:48:24.888736 16552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:53045
I20260619 06:48:24.889353 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 16905
I20260619 06:48:24.889468 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:24.889788 14803 external_mini_cluster.cc:1428] Setting key 47a8f9c81542b2550f75dac004c1ce66
I20260619 06:48:24.891728 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:48:24.896440 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.888564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:55193 (local address 127.14.116.254:41549)
0619 06:48:24.888676 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.888679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.889189 (+ 510us) server_negotiation.cc:407] Connection header received
0619 06:48:24.889326 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.889329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.889370 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.889466 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:24.889961 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.890703 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.891626 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.891922 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.892028 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.893053 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:24.893056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:24.893058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:24.893080 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:24.894747 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.895211 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.895213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.895214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.895285 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.895623 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.895627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.895628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.895790 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:24.895866 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.896168 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.896272 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:48:24.896530 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.888501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:41549 (local address 127.14.116.195:55193)
0619 06:48:24.889099 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:24.889112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:24.889207 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:24.889573 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:24.889576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:24.889739 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:24.889854 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.889863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.890815 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.890819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.891516 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.891525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.892027 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.892031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.892118 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.892607 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:24.892627 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:24.892918 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:24.894849 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.894855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.894860 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.895118 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.895376 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:24.895382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:24.895386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:24.895485 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:24.895891 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:24.895897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:24.896005 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 06:48:24.896126 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 06:48:24.896286 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":74,"threads_started":1}
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:48:24 dist-test-slave-9x3b krb5kdc[16508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:24.916612 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.909466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.14.116.254:41549)
0619 06:48:24.909581 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.909584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.909596 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:48:24.909629 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.909631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.909664 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.909729 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:24.910351 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.910838 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.911590 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.911845 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.911923 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.913999 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:24.914002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:24.914004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:24.914024 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:24.915163 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.915544 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.915546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.915547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.915662 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:24.915897 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:24.915899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:24.915900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:24.916050 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:24.916145 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.916322 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.916411 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:48:24.919147 16552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:24.922302 16552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:24.936352 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.930016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34567 (local address 127.0.0.1:48716)
0619 06:48:24.930465 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:24.930511 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:24.930767 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:24.932053 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:24.932058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:24.932076 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:24.932155 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.932164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.933355 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.933362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.934958 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.934969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.935772 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.935775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.935896 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.935965 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 06:48:24.936067 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:48:24.936198 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":249,"thread_start_us":93,"threads_started":1}
I20260619 06:48:24.936372 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.930412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.14.116.194:46333)
0619 06:48:24.931177 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.931189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.931213 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:48:24.931290 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.931295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.931499 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.931989 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:24.932123 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.933338 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.935081 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.935858 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.935996 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.936113 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.936168 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.936247 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":98,"threads_started":1}
I20260619 06:48:24.936362 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.930190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:46333 (local address 127.0.0.1:48054)
0619 06:48:24.930679 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:24.930694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:24.930767 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:24.931692 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:24.931698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:24.931720 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:24.931842 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.931848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.933447 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.933450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.934968 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.934974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.935966 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.935970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.936062 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.936083 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:48:24.936147 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 06:48:24.936212 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":113,"thread_start_us":114,"threads_started":1}
I20260619 06:48:24.936429 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.931136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.14.116.193:39291)
0619 06:48:24.931476 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.931491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.931513 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:48:24.931584 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.931590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.931809 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.931964 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:24.932214 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.933329 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.934844 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.935586 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.935745 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.935873 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.936191 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.936285 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
I20260619 06:48:24.936708 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.930727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:39291 (local address 127.0.0.1:51676)
0619 06:48:24.931250 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:24.931264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:24.931342 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:24.931931 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:24.931934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:24.931948 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:24.932100 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.932107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.933477 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.933480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.934723 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:24.934732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.935713 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.935719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:24.935842 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.935965 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 06:48:24.936069 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:48:24.936584 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":558,"thread_start_us":84,"threads_started":1}
I20260619 06:48:24.937120 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:24.930739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48716 (local address 127.14.116.195:34567)
0619 06:48:24.931589 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:24.931602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:24.931644 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:48:24.931718 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:24.931725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:24.931915 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:24.932287 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:24.932390 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.933252 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.935099 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:24.935598 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:24.936703 (+ 1105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:24.936846 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:24.936908 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 06:48:24.936993 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":102,"threads_started":1}
I20260619 06:48:24.939390 16690 tablet_service.cc:1467] Processing CreateTablet for tablet 947a7c040c7348769f4a61ccd9e86b16 (DEFAULT_TABLE table=test-table [id=abbbad1ecb524f4c943cf2a020e79eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:24.939505 16830 tablet_service.cc:1467] Processing CreateTablet for tablet 947a7c040c7348769f4a61ccd9e86b16 (DEFAULT_TABLE table=test-table [id=abbbad1ecb524f4c943cf2a020e79eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:24.939505 16970 tablet_service.cc:1467] Processing CreateTablet for tablet 947a7c040c7348769f4a61ccd9e86b16 (DEFAULT_TABLE table=test-table [id=abbbad1ecb524f4c943cf2a020e79eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:24.940402 16690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947a7c040c7348769f4a61ccd9e86b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:24.940495 16970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947a7c040c7348769f4a61ccd9e86b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:24.940487 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947a7c040c7348769f4a61ccd9e86b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:24.946480 17059 tablet_bootstrap.cc:492] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Bootstrap starting.
I20260619 06:48:24.947674 17060 tablet_bootstrap.cc:492] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: Bootstrap starting.
I20260619 06:48:24.948484 17061 tablet_bootstrap.cc:492] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: Bootstrap starting.
I20260619 06:48:24.948901 17059 tablet_bootstrap.cc:654] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:24.949648 17060 tablet_bootstrap.cc:654] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:24.949782 17059 log.cc:824] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:24.950255 17061 tablet_bootstrap.cc:654] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:24.950402 17060 log.cc:824] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:24.951050 17061 log.cc:824] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:24.951912 17059 tablet_bootstrap.cc:492] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: No bootstrap required, opened a new log
I20260619 06:48:24.952111 17059 ts_tablet_manager.cc:1397] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:48:24.952188 17060 tablet_bootstrap.cc:492] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: No bootstrap required, opened a new log
I20260619 06:48:24.952342 17060 ts_tablet_manager.cc:1397] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:48:24.952801 17061 tablet_bootstrap.cc:492] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: No bootstrap required, opened a new log
I20260619 06:48:24.952984 17061 ts_tablet_manager.cc:1397] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:48:24.955955 17061 raft_consensus.cc:348] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } }
I20260619 06:48:24.955977 17059 raft_consensus.cc:348] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } }
I20260619 06:48:24.955976 17060 raft_consensus.cc:348] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } }
I20260619 06:48:24.956173 17061 raft_consensus.cc:374] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:24.956209 17060 raft_consensus.cc:374] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:24.956209 17059 raft_consensus.cc:374] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:24.956218 17061 raft_consensus.cc:729] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd795e0e60e94a67838e492f67e31ef9, State: Initialized, Role: FOLLOWER
I20260619 06:48:24.956257 17059 raft_consensus.cc:729] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c76bf0c82024ff99ea0d14d9eaa4e0b, State: Initialized, Role: FOLLOWER
I20260619 06:48:24.956257 17060 raft_consensus.cc:729] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ec28deed947fea13fae313cc3ba0e, State: Initialized, Role: FOLLOWER
I20260619 06:48:24.956646 17061 consensus_queue.cc:260] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } }
I20260619 06:48:24.956684 17060 consensus_queue.cc:260] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } }
I20260619 06:48:24.956684 17059 consensus_queue.cc:260] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } }
I20260619 06:48:24.957223 16896 heartbeater.cc:503] Master 127.14.116.254:41549 was elected leader, sending a full tablet report...
I20260619 06:48:24.957564 17061 ts_tablet_manager.cc:1428] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:48:24.957567 17059 ts_tablet_manager.cc:1428] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:48:24.957567 17060 ts_tablet_manager.cc:1428] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:48:24.957870 17036 heartbeater.cc:503] Master 127.14.116.254:41549 was elected leader, sending a full tablet report...
I20260619 06:48:24.957868 16756 heartbeater.cc:503] Master 127.14.116.254:41549 was elected leader, sending a full tablet report...
W20260619 06:48:24.959656 16757 tablet.cc:2369] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:25.133563 17066 raft_consensus.cc:482] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260619 06:48:25.133759 17037 tablet.cc:2369] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:25.133796 17066 raft_consensus.cc:504] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } }
I20260619 06:48:25.134960 17066 leader_election.cc:288] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333), 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291)
I20260619 06:48:25.138732 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.135175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:46333 (local address 127.14.116.195:32827)
0619 06:48:25.135290 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:25.135307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:25.135400 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:25.135732 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:25.135737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:25.135757 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:25.135869 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.135874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.136677 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.136679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.137716 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.137725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.138345 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.138347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.138422 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.138443 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:48:25.138532 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 06:48:25.138618 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 06:48:25.138736 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.135273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:32827 (local address 127.14.116.194:46333)
0619 06:48:25.135382 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.135386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.135400 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:25.135535 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.135539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.135595 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.135722 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:25.135980 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.136577 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.137823 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.138213 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.138353 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.138413 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.138546 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.138613 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:48:25.139535 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.135390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:43893 (local address 127.14.116.193:39291)
0619 06:48:25.135513 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.135518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.135658 (+ 140us) server_negotiation.cc:407] Connection header received
0619 06:48:25.135815 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.135818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.135866 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.135959 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:25.136497 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.137042 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.138320 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.138970 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.139081 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.139125 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.139392 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.139448 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:48:25.139492 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a7c040c7348769f4a61ccd9e86b16" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd795e0e60e94a67838e492f67e31ef9" is_pre_election: true
I20260619 06:48:25.139588 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.135182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:39291 (local address 127.14.116.195:43893)
0619 06:48:25.135563 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:25.135573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:25.135706 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:25.135965 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:25.135968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:25.135986 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:25.136377 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.136382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.137143 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.137147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.138202 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.138209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.139155 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.139158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.139270 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.139294 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:48:25.139367 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 06:48:25.139436 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":32,"thread_start_us":127,"threads_started":1}
I20260619 06:48:25.139828 16850 raft_consensus.cc:2454] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 0.
I20260619 06:48:25.140172 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a7c040c7348769f4a61ccd9e86b16" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" is_pre_election: true
I20260619 06:48:25.140349 16921 leader_election.cc:302] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, cd795e0e60e94a67838e492f67e31ef9; no voters:
I20260619 06:48:25.140509 16710 raft_consensus.cc:2454] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 0.
I20260619 06:48:25.140659 17066 raft_consensus.cc:2781] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:25.140734 17066 raft_consensus.cc:482] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:25.140784 17066 raft_consensus.cc:3037] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:25.141813 17066 raft_consensus.cc:504] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } }
I20260619 06:48:25.142198 17066 leader_election.cc:288] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 election: Requested vote from peers cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333), 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291)
I20260619 06:48:25.142473 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a7c040c7348769f4a61ccd9e86b16" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd795e0e60e94a67838e492f67e31ef9"
I20260619 06:48:25.142596 16850 raft_consensus.cc:3037] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:25.142623 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a7c040c7348769f4a61ccd9e86b16" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b"
I20260619 06:48:25.142763 16710 raft_consensus.cc:3037] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:25.143601 16850 raft_consensus.cc:2454] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 1.
I20260619 06:48:25.143803 16710 raft_consensus.cc:2454] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 1.
I20260619 06:48:25.144029 16921 leader_election.cc:302] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, cd795e0e60e94a67838e492f67e31ef9; no voters:
I20260619 06:48:25.144232 17066 raft_consensus.cc:2781] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:25.144390 17066 raft_consensus.cc:686] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 1 LEADER]: Becoming Leader. State: Replica: 445ec28deed947fea13fae313cc3ba0e, State: Running, Role: LEADER
I20260619 06:48:25.144687 17066 consensus_queue.cc:237] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } }
I20260619 06:48:25.147758 16553 catalog_manager.cc:5495] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 445ec28deed947fea13fae313cc3ba0e (127.14.116.195). New cstate: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } health_report { overall_health: UNKNOWN } } }
W20260619 06:48:25.181177 16897 tablet.cc:2369] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:48:25 dist-test-slave-9x3b krb5kdc[16508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851704, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:25.204749 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.197070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.14.116.254:41549)
0619 06:48:25.197196 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.197201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.197349 (+ 148us) server_negotiation.cc:407] Connection header received
0619 06:48:25.197512 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.197515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.197557 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.197637 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:25.198265 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.198778 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.199542 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.199813 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.199932 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.201976 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:25.201979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:25.201981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:25.202001 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:25.203282 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:25.203740 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:25.203742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:25.203743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:25.203810 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:25.204037 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:25.204039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:25.204040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:25.204190 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:25.204260 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.204525 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.204634 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":44}
I20260619 06:48:25.204749 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.196997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:41549 (local address 127.0.0.1:45584)
0619 06:48:25.197264 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:25.197276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:25.197420 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:25.197652 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:25.197654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:25.198062 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:25.198145 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.198151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.198879 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.198882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.199432 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.199439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.199933 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.199936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.200013 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.200490 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:25.200501 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:25.201857 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:25.203376 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:25.203383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:25.203388 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:25.203634 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:25.203886 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:25.203888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:25.203889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:25.203966 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:25.204278 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:25.204283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:25.204381 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 06:48:25.204474 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:48:25.204582 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260619 06:48:25.208293 16553 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:45584:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:48:25.208951 16553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:25.213423 16690 tablet_service.cc:1467] Processing CreateTablet for tablet a48706cc5d874df6b3c9580638665966 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4b03cb9905443eb9e49c1a1fa9b647e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:48:25.213490 16830 tablet_service.cc:1467] Processing CreateTablet for tablet a48706cc5d874df6b3c9580638665966 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4b03cb9905443eb9e49c1a1fa9b647e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:48:25.214140 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48706cc5d874df6b3c9580638665966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:25.214140 16690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48706cc5d874df6b3c9580638665966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:25.213822 16970 tablet_service.cc:1467] Processing CreateTablet for tablet a48706cc5d874df6b3c9580638665966 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4b03cb9905443eb9e49c1a1fa9b647e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:48:25.214504 16970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48706cc5d874df6b3c9580638665966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:25.217032 17061 tablet_bootstrap.cc:492] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: Bootstrap starting.
I20260619 06:48:25.217162 17060 tablet_bootstrap.cc:492] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: Bootstrap starting.
I20260619 06:48:25.217805 17061 tablet_bootstrap.cc:654] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:25.217991 17060 tablet_bootstrap.cc:654] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:25.218340 17059 tablet_bootstrap.cc:492] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Bootstrap starting.
I20260619 06:48:25.218803 17061 tablet_bootstrap.cc:492] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: No bootstrap required, opened a new log
I20260619 06:48:25.218859 17061 ts_tablet_manager.cc:1397] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:25.219067 17059 tablet_bootstrap.cc:654] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:25.219188 17061 raft_consensus.cc:348] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.219264 17061 raft_consensus.cc:374] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:25.219285 17061 raft_consensus.cc:729] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd795e0e60e94a67838e492f67e31ef9, State: Initialized, Role: FOLLOWER
I20260619 06:48:25.219374 17061 consensus_queue.cc:260] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.219653 17061 ts_tablet_manager.cc:1428] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:48:25.219936 17065 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.220074 17065 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:25.220229 17059 tablet_bootstrap.cc:492] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: No bootstrap required, opened a new log
I20260619 06:48:25.220345 17059 ts_tablet_manager.cc:1397] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:25.220747 17059 raft_consensus.cc:348] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.220876 17059 raft_consensus.cc:374] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:25.220916 17059 raft_consensus.cc:729] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c76bf0c82024ff99ea0d14d9eaa4e0b, State: Initialized, Role: FOLLOWER
I20260619 06:48:25.221063 17059 consensus_queue.cc:260] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.221343 17059 ts_tablet_manager.cc:1428] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:25.221385 17060 tablet_bootstrap.cc:492] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: No bootstrap required, opened a new log
I20260619 06:48:25.221356 17067 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.221464 17060 ts_tablet_manager.cc:1397] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 06:48:25.221515 17067 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:25.221896 17060 raft_consensus.cc:348] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.221997 17060 raft_consensus.cc:374] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:25.222030 17060 raft_consensus.cc:729] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ec28deed947fea13fae313cc3ba0e, State: Initialized, Role: FOLLOWER
I20260619 06:48:25.222152 17060 consensus_queue.cc:260] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.222410 17060 ts_tablet_manager.cc:1428] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:48:25.222734 17066 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.222872 17066 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:25.274458 17066 raft_consensus.cc:482] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:25.274580 17066 raft_consensus.cc:504] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.275053 17066 leader_election.cc:288] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291), cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333)
I20260619 06:48:25.275532 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48706cc5d874df6b3c9580638665966" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd795e0e60e94a67838e492f67e31ef9" is_pre_election: true
I20260619 06:48:25.275555 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48706cc5d874df6b3c9580638665966" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" is_pre_election: true
I20260619 06:48:25.275730 16850 raft_consensus.cc:2454] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 0.
I20260619 06:48:25.275748 16710 raft_consensus.cc:2454] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 0.
I20260619 06:48:25.276067 16923 leader_election.cc:302] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, 9c76bf0c82024ff99ea0d14d9eaa4e0b; no voters:
I20260619 06:48:25.276257 17066 raft_consensus.cc:2781] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:25.276350 17066 raft_consensus.cc:482] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:25.276400 17066 raft_consensus.cc:3037] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:25.277423 17066 raft_consensus.cc:504] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.277755 17066 leader_election.cc:288] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 election: Requested vote from peers 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291), cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333)
I20260619 06:48:25.278046 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48706cc5d874df6b3c9580638665966" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b"
I20260619 06:48:25.278083 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48706cc5d874df6b3c9580638665966" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd795e0e60e94a67838e492f67e31ef9"
I20260619 06:48:25.278167 16710 raft_consensus.cc:3037] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:25.278196 16850 raft_consensus.cc:3037] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:25.279121 16710 raft_consensus.cc:2454] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 1.
I20260619 06:48:25.279237 16850 raft_consensus.cc:2454] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 1.
I20260619 06:48:25.279376 16923 leader_election.cc:302] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, 9c76bf0c82024ff99ea0d14d9eaa4e0b; no voters:
I20260619 06:48:25.279538 17066 raft_consensus.cc:2781] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:25.279657 17066 raft_consensus.cc:686] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [term 1 LEADER]: Becoming Leader. State: Replica: 445ec28deed947fea13fae313cc3ba0e, State: Running, Role: LEADER
I20260619 06:48:25.279809 17066 consensus_queue.cc:237] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.280431 17069 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: TxnStatusTablet state changed. Reason: New leader 445ec28deed947fea13fae313cc3ba0e. Latest consensus state: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.280543 17069 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:48:25.280835 17081 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:48:25.281081 17081 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:48:25.281411 16552 catalog_manager.cc:5495] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 445ec28deed947fea13fae313cc3ba0e (127.14.116.195). New cstate: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } health_report { overall_health: HEALTHY } } }
I20260619 06:48:25.301698 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.297794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34567 (local address 127.0.0.1:48728)
0619 06:48:25.297958 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:25.297973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:25.298070 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:25.298373 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:25.298375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:25.298381 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:25.298472 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.298476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.299449 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.299452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.300101 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.300109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.300657 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.300660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.300726 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.300765 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.301464 (+ 699us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.301473 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:48:25.301523 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 06:48:25.301592 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 06:48:25.301716 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.297895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.14.116.195:34567)
0619 06:48:25.298021 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.298025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.298041 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:48:25.298199 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.298202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.298262 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.298360 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:25.298569 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.299334 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.300202 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.300555 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.300655 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.300887 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.301363 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.301477 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.301536 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.301591 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:48:25.306298 16710 raft_consensus.cc:1261] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Refusing update from remote peer 445ec28deed947fea13fae313cc3ba0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:25.306298 16850 raft_consensus.cc:1261] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Refusing update from remote peer 445ec28deed947fea13fae313cc3ba0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:25.306990 17066 consensus_queue.cc:1035] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:25.307197 17069 consensus_queue.cc:1035] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:25.310449 17065 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: TxnStatusTablet state changed. Reason: New leader 445ec28deed947fea13fae313cc3ba0e. Latest consensus state: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.310556 17065 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:25.311188 17067 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: TxnStatusTablet state changed. Reason: New leader 445ec28deed947fea13fae313cc3ba0e. Latest consensus state: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.311298 17067 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:25.314049 17066 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.314152 17066 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:48:25.314275 17066 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.314343 17066 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P 445ec28deed947fea13fae313cc3ba0e: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:48:25.314787 17067 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.314904 17067 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:25.316234 17085 mvcc.cc:205] Tried to move back new op lower bound from 7298464584928104448 to 7298464584827252736. Current Snapshot: MvccSnapshot[applied={T|T < 7298464584928104448}]
I20260619 06:48:25.316396 17086 mvcc.cc:205] Tried to move back new op lower bound from 7298464584928104448 to 7298464584827252736. Current Snapshot: MvccSnapshot[applied={T|T < 7298464584928104448}]
I20260619 06:48:25.316543 17065 tablet_replica.cc:439] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } } }
I20260619 06:48:25.316623 17065 tablet_replica.cc:442] T a48706cc5d874df6b3c9580638665966 P cd795e0e60e94a67838e492f67e31ef9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:25.324823 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.321576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.14.116.195:34567)
0619 06:48:25.321690 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.321694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.321708 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:25.321832 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.321835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.321887 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.321966 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:25.322159 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.322909 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.323767 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.324105 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.324198 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.324439 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.324515 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.324583 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.324657 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.324702 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:48:25.334870 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.331653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:49663 (local address 127.14.116.195:34567)
0619 06:48:25.331758 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.331761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.331773 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:48:25.331845 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.331847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.331881 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.331980 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:25.332200 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.332739 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.333922 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.334116 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.334192 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.334515 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.334568 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.334615 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.334722 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.334758 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:48:25.334967 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.331560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34567 (local address 127.14.116.195:49663)
0619 06:48:25.331688 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:25.331703 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:25.331782 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:25.331994 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:25.331997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:25.332003 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:25.332106 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.332112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.332826 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.332829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.333820 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.333830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.334212 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.334215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.334320 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.334366 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.334640 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.334646 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:48:25.334709 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 06:48:25.334778 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 06:48:25.341163 16710 raft_consensus.cc:1261] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Refusing update from remote peer 445ec28deed947fea13fae313cc3ba0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:25.341171 16850 raft_consensus.cc:1261] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Refusing update from remote peer 445ec28deed947fea13fae313cc3ba0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:25.341559 17089 consensus_queue.cc:1035] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:25.341713 17066 consensus_queue.cc:1035] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:25.346464 17084 mvcc.cc:205] Tried to move back new op lower bound from 7298464585074323456 to 7298464584275226624. Current Snapshot: MvccSnapshot[applied={T|T < 7298464585074323456}]
I20260619 06:48:25.381994 16553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:25.382180 16553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:25.384518 16553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33: Sending DeleteTablet for 3 replicas of tablet 947a7c040c7348769f4a61ccd9e86b16
I20260619 06:48:25.385249 16830 tablet_service.cc:1514] Processing DeleteTablet for tablet 947a7c040c7348769f4a61ccd9e86b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:25 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48054
I20260619 06:48:25.385269 16690 tablet_service.cc:1514] Processing DeleteTablet for tablet 947a7c040c7348769f4a61ccd9e86b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:25 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:51676
I20260619 06:48:25.385747 17101 tablet_replica.cc:330] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: stopping tablet replica
I20260619 06:48:25.386154 17101 raft_consensus.cc:2229] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:25.386129 16970 tablet_service.cc:1514] Processing DeleteTablet for tablet 947a7c040c7348769f4a61ccd9e86b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:25 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48716
I20260619 06:48:25.386396 17101 raft_consensus.cc:2258] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:25.386364 16553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:48:25.386581 17102 tablet_replica.cc:330] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: stopping tablet replica
W20260619 06:48:25.386741 16553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:25.386848 17102 raft_consensus.cc:2229] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:25.386969 17100 tablet_replica.cc:330] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: stopping tablet replica
I20260619 06:48:25.387178 17102 raft_consensus.cc:2258] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:25.387303 17100 raft_consensus.cc:2229] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:25.387509 17100 raft_consensus.cc:2258] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:25.388280 17101 ts_tablet_manager.cc:1905] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:25.388736 17100 ts_tablet_manager.cc:1905] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:25.388865 17102 ts_tablet_manager.cc:1905] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:25.391338 17101 ts_tablet_manager.cc:1918] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:25.391438 17101 log.cc:1196] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/947a7c040c7348769f4a61ccd9e86b16
I20260619 06:48:25.391714 17100 ts_tablet_manager.cc:1918] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:25.391788 17101 ts_tablet_manager.cc:1939] T 947a7c040c7348769f4a61ccd9e86b16 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Deleting consensus metadata
I20260619 06:48:25.391893 17100 log.cc:1196] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/947a7c040c7348769f4a61ccd9e86b16
I20260619 06:48:25.392253 17100 ts_tablet_manager.cc:1939] T 947a7c040c7348769f4a61ccd9e86b16 P cd795e0e60e94a67838e492f67e31ef9: Deleting consensus metadata
I20260619 06:48:25.392405 16690 tablet_service.cc:1467] Processing CreateTablet for tablet 672573e5094144bcbc1fb48ada265a3c (DEFAULT_TABLE table=test-table [id=16a27bfa041543c5afe138ad0ade0513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:25.392472 17102 ts_tablet_manager.cc:1918] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:25.392547 17102 log.cc:1196] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/947a7c040c7348769f4a61ccd9e86b16
I20260619 06:48:25.392642 16690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672573e5094144bcbc1fb48ada265a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:25.392833 17102 ts_tablet_manager.cc:1939] T 947a7c040c7348769f4a61ccd9e86b16 P 445ec28deed947fea13fae313cc3ba0e: Deleting consensus metadata
I20260619 06:48:25.394726 17059 tablet_bootstrap.cc:492] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Bootstrap starting.
I20260619 06:48:25.394901 16970 tablet_service.cc:1467] Processing CreateTablet for tablet 672573e5094144bcbc1fb48ada265a3c (DEFAULT_TABLE table=test-table [id=16a27bfa041543c5afe138ad0ade0513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:25.395128 16970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672573e5094144bcbc1fb48ada265a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:25.395702 17059 tablet_bootstrap.cc:654] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:25.396727 17059 tablet_bootstrap.cc:492] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: No bootstrap required, opened a new log
I20260619 06:48:25.396822 17059 ts_tablet_manager.cc:1397] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:48:25.397109 17060 tablet_bootstrap.cc:492] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: Bootstrap starting.
I20260619 06:48:25.397254 17059 raft_consensus.cc:348] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.397348 17059 raft_consensus.cc:374] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:25.397379 17059 raft_consensus.cc:729] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c76bf0c82024ff99ea0d14d9eaa4e0b, State: Initialized, Role: FOLLOWER
I20260619 06:48:25.397492 17059 consensus_queue.cc:260] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.397516 16539 catalog_manager.cc:4841] TS 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291): tablet 947a7c040c7348769f4a61ccd9e86b16 (table test-table [id=abbbad1ecb524f4c943cf2a020e79eb8]) successfully deleted
I20260619 06:48:25.397719 17059 ts_tablet_manager.cc:1428] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:48:25.397725 16539 catalog_manager.cc:4841] TS 445ec28deed947fea13fae313cc3ba0e (127.14.116.195:34567): tablet 947a7c040c7348769f4a61ccd9e86b16 (table test-table [id=abbbad1ecb524f4c943cf2a020e79eb8]) successfully deleted
I20260619 06:48:25.397881 17060 tablet_bootstrap.cc:654] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:25.398160 16537 catalog_manager.cc:4841] TS cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333): tablet 947a7c040c7348769f4a61ccd9e86b16 (table test-table [id=abbbad1ecb524f4c943cf2a020e79eb8]) successfully deleted
I20260619 06:48:25.398746 16830 tablet_service.cc:1467] Processing CreateTablet for tablet 672573e5094144bcbc1fb48ada265a3c (DEFAULT_TABLE table=test-table [id=16a27bfa041543c5afe138ad0ade0513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:25.398983 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672573e5094144bcbc1fb48ada265a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:25.399875 17060 tablet_bootstrap.cc:492] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: No bootstrap required, opened a new log
I20260619 06:48:25.399981 17060 ts_tablet_manager.cc:1397] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 06:48:25.400413 17060 raft_consensus.cc:348] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.400542 17060 raft_consensus.cc:374] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:25.400578 17060 raft_consensus.cc:729] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ec28deed947fea13fae313cc3ba0e, State: Initialized, Role: FOLLOWER
I20260619 06:48:25.400692 17060 consensus_queue.cc:260] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.400954 17060 ts_tablet_manager.cc:1428] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:48:25.401125 17061 tablet_bootstrap.cc:492] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: Bootstrap starting.
I20260619 06:48:25.401947 17061 tablet_bootstrap.cc:654] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:25.402961 17061 tablet_bootstrap.cc:492] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: No bootstrap required, opened a new log
I20260619 06:48:25.403054 17061 ts_tablet_manager.cc:1397] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:25.403388 17061 raft_consensus.cc:348] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.403462 17061 raft_consensus.cc:374] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:25.403483 17061 raft_consensus.cc:729] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd795e0e60e94a67838e492f67e31ef9, State: Initialized, Role: FOLLOWER
I20260619 06:48:25.403601 17061 consensus_queue.cc:260] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.403875 17061 ts_tablet_manager.cc:1428] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:25.415879 17065 raft_consensus.cc:482] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:25.415994 17065 raft_consensus.cc:504] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.417019 17065 leader_election.cc:288] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291), 445ec28deed947fea13fae313cc3ba0e (127.14.116.195:34567)
I20260619 06:48:25.421137 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.417662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:43817 (local address 127.14.116.195:34567)
0619 06:48:25.417788 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.417793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.417961 (+ 168us) server_negotiation.cc:407] Connection header received
0619 06:48:25.418079 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.418083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.418126 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.418200 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:25.418498 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.419014 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.420141 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.420575 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.420688 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.420739 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.420955 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.421020 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:48:25.421142 17107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.417439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34567 (local address 127.14.116.194:43817)
0619 06:48:25.417877 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:25.417887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:25.417997 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:25.418219 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:25.418221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:25.418236 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:25.418351 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.418356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.419124 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.419127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.420023 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.420031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.420708 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.420712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.420814 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.420841 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:48:25.420925 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:48:25.420990 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":91,"threads_started":1}
I20260619 06:48:25.421748 16988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672573e5094144bcbc1fb48ada265a3c" candidate_uuid: "cd795e0e60e94a67838e492f67e31ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ec28deed947fea13fae313cc3ba0e" is_pre_election: true
I20260619 06:48:25.421870 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.417354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:54107 (local address 127.14.116.193:39291)
0619 06:48:25.417495 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.417502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.417602 (+ 100us) server_negotiation.cc:407] Connection header received
0619 06:48:25.417790 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.417794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.417845 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.417893 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:25.418250 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.419064 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.420541 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.421112 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.421228 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.421282 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.421701 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.421765 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:25.421964 16988 raft_consensus.cc:2454] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd795e0e60e94a67838e492f67e31ef9 in term 0.
I20260619 06:48:25.422205 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.417155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:39291 (local address 127.14.116.194:54107)
0619 06:48:25.417500 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:25.417519 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:25.417672 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:25.418027 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:25.418030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:25.418046 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:25.418142 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.418147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.419207 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.419210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.420408 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.420419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.421480 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.421486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.421572 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.421599 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:48:25.421673 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:48:25.421733 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":60,"threads_started":1}
I20260619 06:48:25.422401 16783 leader_election.cc:302] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, cd795e0e60e94a67838e492f67e31ef9; no voters:
I20260619 06:48:25.422684 17065 raft_consensus.cc:2781] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:25.422742 17065 raft_consensus.cc:482] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:25.422789 17065 raft_consensus.cc:3037] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:25.422945 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672573e5094144bcbc1fb48ada265a3c" candidate_uuid: "cd795e0e60e94a67838e492f67e31ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" is_pre_election: true
I20260619 06:48:25.423081 16710 raft_consensus.cc:2454] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd795e0e60e94a67838e492f67e31ef9 in term 0.
I20260619 06:48:25.424007 17065 raft_consensus.cc:504] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.424388 17065 leader_election.cc:288] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [CANDIDATE]: Term 1 election: Requested vote from peers 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291), 445ec28deed947fea13fae313cc3ba0e (127.14.116.195:34567)
I20260619 06:48:25.424665 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672573e5094144bcbc1fb48ada265a3c" candidate_uuid: "cd795e0e60e94a67838e492f67e31ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b"
I20260619 06:48:25.424796 16710 raft_consensus.cc:3037] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:25.424804 16988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672573e5094144bcbc1fb48ada265a3c" candidate_uuid: "cd795e0e60e94a67838e492f67e31ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ec28deed947fea13fae313cc3ba0e"
I20260619 06:48:25.424916 16988 raft_consensus.cc:3037] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:25.425987 16988 raft_consensus.cc:2454] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd795e0e60e94a67838e492f67e31ef9 in term 1.
I20260619 06:48:25.425981 16710 raft_consensus.cc:2454] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd795e0e60e94a67838e492f67e31ef9 in term 1.
I20260619 06:48:25.426229 16783 leader_election.cc:302] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, cd795e0e60e94a67838e492f67e31ef9; no voters:
I20260619 06:48:25.426404 17065 raft_consensus.cc:2781] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:25.426631 17065 raft_consensus.cc:686] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 1 LEADER]: Becoming Leader. State: Replica: cd795e0e60e94a67838e492f67e31ef9, State: Running, Role: LEADER
I20260619 06:48:25.426856 17065 consensus_queue.cc:237] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:25.429185 16553 catalog_manager.cc:5495] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194). New cstate: current_term: 1 leader_uuid: "cd795e0e60e94a67838e492f67e31ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } health_report { overall_health: UNKNOWN } } }
I20260619 06:48:25.450697 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.446911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.14.116.194:46333)
0619 06:48:25.447018 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.447022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.447089 (+ 67us) server_negotiation.cc:407] Connection header received
0619 06:48:25.447224 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.447227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.447285 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.447406 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:25.447586 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.448395 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.449165 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.449501 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.449602 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.449796 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.450349 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.450469 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.450503 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.450554 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:48:25.461597 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.457732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:46621 (local address 127.14.116.195:34567)
0619 06:48:25.457862 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.457867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.457882 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:48:25.457940 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.457943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.457991 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.458066 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:25.458347 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.459089 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.460288 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.460675 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.460766 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.461109 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.461186 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.461254 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.461413 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.461466 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:48:25.461910 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.457405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34567 (local address 127.14.116.194:46621)
0619 06:48:25.457707 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:25.457721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:25.457845 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:25.458087 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:25.458090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:25.458097 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:25.458222 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.458231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.459252 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.459255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.460151 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.460161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.460839 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.460842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.460933 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.460976 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.461292 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.461298 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:48:25.461359 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 06:48:25.461438 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":135,"threads_started":1}
I20260619 06:48:25.468897 16988 raft_consensus.cc:1261] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Refusing update from remote peer cd795e0e60e94a67838e492f67e31ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:25.469324 16710 raft_consensus.cc:1261] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Refusing update from remote peer cd795e0e60e94a67838e492f67e31ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:25.469448 17108 consensus_queue.cc:1035] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:25.469695 17065 consensus_queue.cc:1035] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:25.490280 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.485888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:46333 (local address 127.14.116.195:45555)
0619 06:48:25.486105 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:25.486121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:25.486198 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:25.487231 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:25.487233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:25.487241 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:25.487341 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.487347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.488268 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.488273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.488860 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:25.488868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.489585 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.489588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:25.489672 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.489700 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.489998 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.490005 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:48:25.490064 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 06:48:25.490150 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260619 06:48:25.490686 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:25.486834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:45555 (local address 127.14.116.194:46333)
0619 06:48:25.487023 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:25.487028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:25.487042 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:25.487080 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:25.487083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:25.487129 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:25.487210 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:25.487464 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.488160 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.489118 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:25.489373 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:25.489499 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:25.489822 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:25.489897 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:25.489961 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:25.490566 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 06:48:25.490607 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260619 06:48:25.491279 17086 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:48:27.046128 16552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:27.046267 16552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:27.048377 16552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33: Sending DeleteTablet for 3 replicas of tablet 672573e5094144bcbc1fb48ada265a3c
I20260619 06:48:27.048864 16830 tablet_service.cc:1514] Processing DeleteTablet for tablet 672573e5094144bcbc1fb48ada265a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:27 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48054
I20260619 06:48:27.048864 16690 tablet_service.cc:1514] Processing DeleteTablet for tablet 672573e5094144bcbc1fb48ada265a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:27 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:51676
I20260619 06:48:27.048913 16970 tablet_service.cc:1514] Processing DeleteTablet for tablet 672573e5094144bcbc1fb48ada265a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:27 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48716
I20260619 06:48:27.049345 17140 tablet_replica.cc:330] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: stopping tablet replica
I20260619 06:48:27.049499 17140 raft_consensus.cc:2229] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:27.049626 17140 raft_consensus.cc:2258] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:27.050277 17140 ts_tablet_manager.cc:1905] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:27.051784 17139 tablet_replica.cc:330] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: stopping tablet replica
I20260619 06:48:27.051981 17138 tablet_replica.cc:330] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: stopping tablet replica
I20260619 06:48:27.052124 17138 raft_consensus.cc:2229] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:27.052227 17138 raft_consensus.cc:2258] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:27.052403 17139 raft_consensus.cc:2229] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:27.052681 17139 raft_consensus.cc:2258] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:27.052971 17140 ts_tablet_manager.cc:1918] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:48:27.053051 17140 log.cc:1196] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/672573e5094144bcbc1fb48ada265a3c
I20260619 06:48:27.053184 17138 ts_tablet_manager.cc:1905] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:27.053314 17140 ts_tablet_manager.cc:1939] T 672573e5094144bcbc1fb48ada265a3c P 445ec28deed947fea13fae313cc3ba0e: Deleting consensus metadata
I20260619 06:48:27.053994 16539 catalog_manager.cc:4841] TS 445ec28deed947fea13fae313cc3ba0e (127.14.116.195:34567): tablet 672573e5094144bcbc1fb48ada265a3c (table test-table [id=16a27bfa041543c5afe138ad0ade0513]) successfully deleted
I20260619 06:48:27.055131 17138 ts_tablet_manager.cc:1918] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:48:27.055217 17138 log.cc:1196] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/672573e5094144bcbc1fb48ada265a3c
I20260619 06:48:27.055470 17138 ts_tablet_manager.cc:1939] T 672573e5094144bcbc1fb48ada265a3c P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Deleting consensus metadata
I20260619 06:48:27.055948 17139 ts_tablet_manager.cc:1905] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:27.056115 16539 catalog_manager.cc:4841] TS 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291): tablet 672573e5094144bcbc1fb48ada265a3c (table test-table [id=16a27bfa041543c5afe138ad0ade0513]) successfully deleted
I20260619 06:48:27.058346 17139 ts_tablet_manager.cc:1918] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:48:27.058414 17139 log.cc:1196] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/672573e5094144bcbc1fb48ada265a3c
I20260619 06:48:27.058615 17139 ts_tablet_manager.cc:1939] T 672573e5094144bcbc1fb48ada265a3c P cd795e0e60e94a67838e492f67e31ef9: Deleting consensus metadata
I20260619 06:48:27.059185 16537 catalog_manager.cc:4841] TS cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333): tablet 672573e5094144bcbc1fb48ada265a3c (table test-table [id=16a27bfa041543c5afe138ad0ade0513]) successfully deleted
Jun 19 06:48:27 dist-test-slave-9x3b krb5kdc[16508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:48:27 dist-test-slave-9x3b krb5kdc[16508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:27.083367 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.072563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.14.116.254:41549)
0619 06:48:27.072802 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:27.072811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:27.072906 (+ 95us) server_negotiation.cc:407] Connection header received
0619 06:48:27.073076 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:27.073079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:27.073125 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:27.073224 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:27.073888 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.074454 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.075280 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.075526 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.075649 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.080565 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:27.080568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:27.080570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:27.080593 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:27.081823 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:27.082174 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:27.082176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:27.082178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:27.082247 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:27.082559 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:27.082564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:27.082565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:27.082800 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:27.082906 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:27.083099 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 06:48:27.083194 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260619 06:48:27.085760 16552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:27.086216 16552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:27.090610 16690 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9a10c4a1c840298ce2dcc3059cabd6 (DEFAULT_TABLE table=test-table [id=0130d2d449694f4db44cbc4c46793380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:27.090735 16830 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9a10c4a1c840298ce2dcc3059cabd6 (DEFAULT_TABLE table=test-table [id=0130d2d449694f4db44cbc4c46793380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:27.090916 16690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9a10c4a1c840298ce2dcc3059cabd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:27.090977 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9a10c4a1c840298ce2dcc3059cabd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:27.091085 16970 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9a10c4a1c840298ce2dcc3059cabd6 (DEFAULT_TABLE table=test-table [id=0130d2d449694f4db44cbc4c46793380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:27.091351 16970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9a10c4a1c840298ce2dcc3059cabd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:27.093317 17149 tablet_bootstrap.cc:492] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: Bootstrap starting.
I20260619 06:48:27.093670 17150 tablet_bootstrap.cc:492] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: Bootstrap starting.
I20260619 06:48:27.094177 17149 tablet_bootstrap.cc:654] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:27.094585 17150 tablet_bootstrap.cc:654] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:27.095187 17151 tablet_bootstrap.cc:492] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Bootstrap starting.
I20260619 06:48:27.095347 17149 tablet_bootstrap.cc:492] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: No bootstrap required, opened a new log
I20260619 06:48:27.095438 17149 ts_tablet_manager.cc:1397] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:27.095827 17150 tablet_bootstrap.cc:492] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: No bootstrap required, opened a new log
I20260619 06:48:27.095923 17150 ts_tablet_manager.cc:1397] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:27.095876 17149 raft_consensus.cc:348] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.095981 17149 raft_consensus.cc:374] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:27.096024 17149 raft_consensus.cc:729] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ec28deed947fea13fae313cc3ba0e, State: Initialized, Role: FOLLOWER
I20260619 06:48:27.096194 17149 consensus_queue.cc:260] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.096335 17150 raft_consensus.cc:348] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.096418 17151 tablet_bootstrap.cc:654] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:27.096443 17150 raft_consensus.cc:374] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:27.096473 17150 raft_consensus.cc:729] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd795e0e60e94a67838e492f67e31ef9, State: Initialized, Role: FOLLOWER
I20260619 06:48:27.096625 17150 consensus_queue.cc:260] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.096899 17150 ts_tablet_manager.cc:1428] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:27.097831 17149 ts_tablet_manager.cc:1428] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:27.098670 17151 tablet_bootstrap.cc:492] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: No bootstrap required, opened a new log
I20260619 06:48:27.099161 17151 ts_tablet_manager.cc:1397] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:48:27.099751 17151 raft_consensus.cc:348] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.099859 17151 raft_consensus.cc:374] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:27.099916 17151 raft_consensus.cc:729] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c76bf0c82024ff99ea0d14d9eaa4e0b, State: Initialized, Role: FOLLOWER
I20260619 06:48:27.100034 17151 consensus_queue.cc:260] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.100358 17151 ts_tablet_manager.cc:1428] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:27.138422 17098 raft_consensus.cc:482] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:27.138533 17098 raft_consensus.cc:504] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.138999 17098 leader_election.cc:288] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291), cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333)
I20260619 06:48:27.139336 16709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9a10c4a1c840298ce2dcc3059cabd6" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" is_pre_election: true
I20260619 06:48:27.139384 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9a10c4a1c840298ce2dcc3059cabd6" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd795e0e60e94a67838e492f67e31ef9" is_pre_election: true
I20260619 06:48:27.139513 16850 raft_consensus.cc:2454] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 0.
I20260619 06:48:27.139513 16709 raft_consensus.cc:2454] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 0.
I20260619 06:48:27.139855 16923 leader_election.cc:302] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, 9c76bf0c82024ff99ea0d14d9eaa4e0b; no voters:
I20260619 06:48:27.140007 17098 raft_consensus.cc:2781] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:27.140064 17098 raft_consensus.cc:482] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:27.140112 17098 raft_consensus.cc:3037] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:27.141016 17098 raft_consensus.cc:504] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.141319 17098 leader_election.cc:288] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 election: Requested vote from peers 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291), cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333)
I20260619 06:48:27.141605 16709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9a10c4a1c840298ce2dcc3059cabd6" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b"
I20260619 06:48:27.141646 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9a10c4a1c840298ce2dcc3059cabd6" candidate_uuid: "445ec28deed947fea13fae313cc3ba0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd795e0e60e94a67838e492f67e31ef9"
I20260619 06:48:27.141721 16709 raft_consensus.cc:3037] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:27.141791 16850 raft_consensus.cc:3037] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:27.142679 16709 raft_consensus.cc:2454] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 1.
I20260619 06:48:27.142792 16850 raft_consensus.cc:2454] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ec28deed947fea13fae313cc3ba0e in term 1.
I20260619 06:48:27.142936 16923 leader_election.cc:302] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, 9c76bf0c82024ff99ea0d14d9eaa4e0b; no voters:
I20260619 06:48:27.143121 17098 raft_consensus.cc:2781] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:27.143257 17098 raft_consensus.cc:686] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 1 LEADER]: Becoming Leader. State: Replica: 445ec28deed947fea13fae313cc3ba0e, State: Running, Role: LEADER
I20260619 06:48:27.143400 17098 consensus_queue.cc:237] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.144984 16552 catalog_manager.cc:5495] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 445ec28deed947fea13fae313cc3ba0e (127.14.116.195). New cstate: current_term: 1 leader_uuid: "445ec28deed947fea13fae313cc3ba0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } health_report { overall_health: HEALTHY } } }
I20260619 06:48:27.165711 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.162898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.14.116.195:34567)
0619 06:48:27.163137 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:27.163143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:27.163155 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:48:27.163188 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:27.163190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:27.163228 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:27.163297 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:27.163513 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.164027 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.164801 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.165009 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.165081 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.165288 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:27.165348 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:27.165412 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:27.165492 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 06:48:27.165530 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
I20260619 06:48:27.171007 16709 raft_consensus.cc:1261] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Refusing update from remote peer 445ec28deed947fea13fae313cc3ba0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:27.171331 16850 raft_consensus.cc:1261] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Refusing update from remote peer 445ec28deed947fea13fae313cc3ba0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:27.171339 17098 consensus_queue.cc:1035] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:27.171738 17098 consensus_queue.cc:1035] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:27.202795 16550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:27.202930 16550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:27.204846 16550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33: Sending DeleteTablet for 3 replicas of tablet 0c9a10c4a1c840298ce2dcc3059cabd6
I20260619 06:48:27.205340 16690 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9a10c4a1c840298ce2dcc3059cabd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:27 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:51676
I20260619 06:48:27.205459 16830 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9a10c4a1c840298ce2dcc3059cabd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:27 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48054
I20260619 06:48:27.205524 17138 tablet_replica.cc:330] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: stopping tablet replica
I20260619 06:48:27.205637 17138 raft_consensus.cc:2229] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:27.205670 16969 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9a10c4a1c840298ce2dcc3059cabd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:27 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48716
I20260619 06:48:27.205778 17138 raft_consensus.cc:2258] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:27.205865 17140 tablet_replica.cc:330] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: stopping tablet replica
I20260619 06:48:27.205942 17139 tablet_replica.cc:330] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: stopping tablet replica
I20260619 06:48:27.205998 17140 raft_consensus.cc:2229] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:27.206043 17139 raft_consensus.cc:2229] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:27.206130 17139 raft_consensus.cc:2258] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:27.206166 17140 raft_consensus.cc:2258] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:27.206456 17138 ts_tablet_manager.cc:1905] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:27.206807 17139 ts_tablet_manager.cc:1905] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:27.206969 17140 ts_tablet_manager.cc:1905] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:27.206965 16550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:27.207337 16550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:27.209064 17138 ts_tablet_manager.cc:1918] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:27.209133 17138 log.cc:1196] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/0c9a10c4a1c840298ce2dcc3059cabd6
I20260619 06:48:27.209312 17139 ts_tablet_manager.cc:1918] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:27.209390 17139 log.cc:1196] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/0c9a10c4a1c840298ce2dcc3059cabd6
I20260619 06:48:27.209415 17138 ts_tablet_manager.cc:1939] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Deleting consensus metadata
I20260619 06:48:27.209502 17140 ts_tablet_manager.cc:1918] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:27.209569 17140 log.cc:1196] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/0c9a10c4a1c840298ce2dcc3059cabd6
I20260619 06:48:27.209664 17139 ts_tablet_manager.cc:1939] T 0c9a10c4a1c840298ce2dcc3059cabd6 P cd795e0e60e94a67838e492f67e31ef9: Deleting consensus metadata
I20260619 06:48:27.209818 17140 ts_tablet_manager.cc:1939] T 0c9a10c4a1c840298ce2dcc3059cabd6 P 445ec28deed947fea13fae313cc3ba0e: Deleting consensus metadata
I20260619 06:48:27.210067 16539 catalog_manager.cc:4841] TS 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291): tablet 0c9a10c4a1c840298ce2dcc3059cabd6 (table test-table [id=0130d2d449694f4db44cbc4c46793380]) successfully deleted
I20260619 06:48:27.210312 16537 catalog_manager.cc:4841] TS cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333): tablet 0c9a10c4a1c840298ce2dcc3059cabd6 (table test-table [id=0130d2d449694f4db44cbc4c46793380]) successfully deleted
I20260619 06:48:27.210681 16539 catalog_manager.cc:4841] TS 445ec28deed947fea13fae313cc3ba0e (127.14.116.195:34567): tablet 0c9a10c4a1c840298ce2dcc3059cabd6 (table test-table [id=0130d2d449694f4db44cbc4c46793380]) successfully deleted
I20260619 06:48:27.211846 16690 tablet_service.cc:1467] Processing CreateTablet for tablet 56ecbe347d254beeb644d967a2fa5a1a (DEFAULT_TABLE table=test-table [id=d146c39ffd2d4f0688d83bc146345849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:27.211853 16830 tablet_service.cc:1467] Processing CreateTablet for tablet 56ecbe347d254beeb644d967a2fa5a1a (DEFAULT_TABLE table=test-table [id=d146c39ffd2d4f0688d83bc146345849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:27.212076 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ecbe347d254beeb644d967a2fa5a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:27.212076 16690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ecbe347d254beeb644d967a2fa5a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:27.214092 17151 tablet_bootstrap.cc:492] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Bootstrap starting.
I20260619 06:48:27.214092 17150 tablet_bootstrap.cc:492] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: Bootstrap starting.
I20260619 06:48:27.214355 16969 tablet_service.cc:1467] Processing CreateTablet for tablet 56ecbe347d254beeb644d967a2fa5a1a (DEFAULT_TABLE table=test-table [id=d146c39ffd2d4f0688d83bc146345849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:27.214653 16969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ecbe347d254beeb644d967a2fa5a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:27.214955 17151 tablet_bootstrap.cc:654] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:27.214955 17150 tablet_bootstrap.cc:654] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:27.216307 17150 tablet_bootstrap.cc:492] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: No bootstrap required, opened a new log
I20260619 06:48:27.216396 17150 ts_tablet_manager.cc:1397] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:27.216430 17151 tablet_bootstrap.cc:492] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b: No bootstrap required, opened a new log
I20260619 06:48:27.216508 17151 ts_tablet_manager.cc:1397] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:27.216801 17150 raft_consensus.cc:348] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.216897 17149 tablet_bootstrap.cc:492] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: Bootstrap starting.
I20260619 06:48:27.216948 17150 raft_consensus.cc:374] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:27.216907 17151 raft_consensus.cc:348] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.216986 17150 raft_consensus.cc:729] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd795e0e60e94a67838e492f67e31ef9, State: Initialized, Role: FOLLOWER
I20260619 06:48:27.217006 17151 raft_consensus.cc:374] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:27.217039 17151 raft_consensus.cc:729] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c76bf0c82024ff99ea0d14d9eaa4e0b, State: Initialized, Role: FOLLOWER
I20260619 06:48:27.217108 17150 consensus_queue.cc:260] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.217144 17151 consensus_queue.cc:260] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.217372 17150 ts_tablet_manager.cc:1428] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:27.217382 17151 ts_tablet_manager.cc:1428] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:27.217741 17149 tablet_bootstrap.cc:654] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:27.218751 17149 tablet_bootstrap.cc:492] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: No bootstrap required, opened a new log
I20260619 06:48:27.218848 17149 ts_tablet_manager.cc:1397] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:27.219183 17149 raft_consensus.cc:348] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.219296 17149 raft_consensus.cc:374] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:27.219336 17149 raft_consensus.cc:729] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ec28deed947fea13fae313cc3ba0e, State: Initialized, Role: FOLLOWER
I20260619 06:48:27.219467 17149 consensus_queue.cc:260] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.219753 17149 ts_tablet_manager.cc:1428] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:27.366381 17155 raft_consensus.cc:482] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:27.366518 17155 raft_consensus.cc:504] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.367693 17155 leader_election.cc:288] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333), 445ec28deed947fea13fae313cc3ba0e (127.14.116.195:34567)
I20260619 06:48:27.372963 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.368200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:51109 (local address 127.14.116.195:34567)
0619 06:48:27.368311 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:27.368315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:27.368372 (+ 57us) server_negotiation.cc:407] Connection header received
0619 06:48:27.368549 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:27.368555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:27.368611 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:27.368688 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:27.368926 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.369705 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.371659 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.372257 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.372395 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.372437 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:27.372766 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 06:48:27.372837 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:48:27.372978 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.368054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:58507 (local address 127.14.116.194:46333)
0619 06:48:27.368284 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:27.368300 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:27.368353 (+ 53us) server_negotiation.cc:407] Connection header received
0619 06:48:27.368416 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:27.368419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:27.368460 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:27.368569 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:27.368865 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.369704 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.371659 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.372266 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.372382 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.372434 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:27.372766 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 06:48:27.372837 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":60,"threads_started":1}
I20260619 06:48:27.373005 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.367897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:46333 (local address 127.14.116.193:58507)
0619 06:48:27.368185 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:27.368200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:27.368305 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:27.368582 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:27.368584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:27.368597 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:27.368744 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:27.368750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.369838 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.369842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:27.371517 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:27.371529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.372382 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.372395 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:27.372570 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.372653 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 06:48:27.372733 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:48:27.372836 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":314,"thread_start_us":81,"threads_started":1}
I20260619 06:48:27.373010 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.367932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34567 (local address 127.14.116.193:51109)
0619 06:48:27.368298 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:27.368311 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:27.368439 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:27.368715 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:27.368718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:27.368731 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:27.368827 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:27.368832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.369838 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.369842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:27.371521 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:27.371529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.372382 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.372386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:27.372598 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.372654 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 06:48:27.372733 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:48:27.372836 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":260,"thread_start_us":99,"threads_started":1}
I20260619 06:48:27.373529 16988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ecbe347d254beeb644d967a2fa5a1a" candidate_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ec28deed947fea13fae313cc3ba0e" is_pre_election: true
I20260619 06:48:27.373699 16988 raft_consensus.cc:2454] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c76bf0c82024ff99ea0d14d9eaa4e0b in term 0.
I20260619 06:48:27.373818 16849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ecbe347d254beeb644d967a2fa5a1a" candidate_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd795e0e60e94a67838e492f67e31ef9" is_pre_election: true
I20260619 06:48:27.373987 16849 raft_consensus.cc:2454] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c76bf0c82024ff99ea0d14d9eaa4e0b in term 0.
I20260619 06:48:27.374126 16643 leader_election.cc:302] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, 9c76bf0c82024ff99ea0d14d9eaa4e0b; no voters:
I20260619 06:48:27.374365 17155 raft_consensus.cc:2781] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:27.374423 17155 raft_consensus.cc:482] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:27.374466 17155 raft_consensus.cc:3037] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:27.376008 17155 raft_consensus.cc:504] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.376359 17155 leader_election.cc:288] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [CANDIDATE]: Term 1 election: Requested vote from peers cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333), 445ec28deed947fea13fae313cc3ba0e (127.14.116.195:34567)
I20260619 06:48:27.376670 16849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ecbe347d254beeb644d967a2fa5a1a" candidate_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd795e0e60e94a67838e492f67e31ef9"
I20260619 06:48:27.376670 16988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ecbe347d254beeb644d967a2fa5a1a" candidate_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ec28deed947fea13fae313cc3ba0e"
I20260619 06:48:27.376784 16988 raft_consensus.cc:3037] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:27.376784 16849 raft_consensus.cc:3037] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:27.377853 16849 raft_consensus.cc:2454] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c76bf0c82024ff99ea0d14d9eaa4e0b in term 1.
I20260619 06:48:27.377853 16988 raft_consensus.cc:2454] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c76bf0c82024ff99ea0d14d9eaa4e0b in term 1.
I20260619 06:48:27.378105 16643 leader_election.cc:302] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ec28deed947fea13fae313cc3ba0e, 9c76bf0c82024ff99ea0d14d9eaa4e0b; no voters:
I20260619 06:48:27.378252 17155 raft_consensus.cc:2781] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:27.378454 17155 raft_consensus.cc:686] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [term 1 LEADER]: Becoming Leader. State: Replica: 9c76bf0c82024ff99ea0d14d9eaa4e0b, State: Running, Role: LEADER
I20260619 06:48:27.378708 17155 consensus_queue.cc:237] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } }
I20260619 06:48:27.380822 16550 catalog_manager.cc:5495] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193). New cstate: current_term: 1 leader_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c76bf0c82024ff99ea0d14d9eaa4e0b" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 } health_report { overall_health: UNKNOWN } } }
I20260619 06:48:27.412222 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.408827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.14.116.193:39291)
0619 06:48:27.409097 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:27.409104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:27.409116 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:48:27.409184 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:27.409186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:27.409237 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:27.409374 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:27.409542 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.410055 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.410828 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.411071 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.411151 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.411365 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:27.411902 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:27.411989 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:27.412049 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 06:48:27.412091 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":125,"threads_started":1}
I20260619 06:48:27.421687 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.418452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:45305 (local address 127.14.116.195:34567)
0619 06:48:27.418573 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:27.418577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:27.418767 (+ 190us) server_negotiation.cc:407] Connection header received
0619 06:48:27.418932 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:27.418936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:27.418989 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:27.419069 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:27.419315 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.419892 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.420710 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.420952 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.421050 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.421298 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:27.421361 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:27.421418 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:27.421543 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 06:48:27.421585 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:48:27.421738 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.418350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34567 (local address 127.14.116.193:45305)
0619 06:48:27.418687 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:27.418698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:27.418823 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:27.419080 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:27.419083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:27.419089 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:27.419197 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:27.419210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.419995 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.420000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:27.420594 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:27.420601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.421056 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.421059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:27.421131 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.421171 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:27.421465 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:27.421471 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:48:27.421539 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:48:27.421604 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260619 06:48:27.427189 16849 raft_consensus.cc:1261] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Refusing update from remote peer 9c76bf0c82024ff99ea0d14d9eaa4e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:27.427187 16988 raft_consensus.cc:1261] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Refusing update from remote peer 9c76bf0c82024ff99ea0d14d9eaa4e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:27.427691 17168 consensus_queue.cc:1035] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd795e0e60e94a67838e492f67e31ef9" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:27.427858 17155 consensus_queue.cc:1035] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "445ec28deed947fea13fae313cc3ba0e" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:27.444650 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.441430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:55213 (local address 127.14.116.193:39291)
0619 06:48:27.441548 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:27.441553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:27.441707 (+ 154us) server_negotiation.cc:407] Connection header received
0619 06:48:27.441862 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:27.441865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:27.441909 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:27.441987 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:27.442219 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.442716 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.443547 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.443860 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.443968 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.444239 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:27.444312 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:27.444390 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:27.444479 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 06:48:27.444525 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:48:27.444685 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:27.441380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:39291 (local address 127.14.116.195:55213)
0619 06:48:27.441616 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:27.441629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:27.441757 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:27.442007 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:27.442009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:27.442017 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:27.442114 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:27.442121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.442828 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.442832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:27.443428 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:27.443436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:27.443968 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:27.443990 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:27.444075 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:27.444109 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:27.444405 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:27.444411 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:48:27.444470 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 06:48:27.444545 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
W20260619 06:48:27.445297 17121 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:48:28.997445 16550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:28.997587 16550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:28.999739 16550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02773ce996d2496189c5a945bd1e1b33: Sending DeleteTablet for 3 replicas of tablet 56ecbe347d254beeb644d967a2fa5a1a
I20260619 06:48:29.000233 16690 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ecbe347d254beeb644d967a2fa5a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:28 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:51676
I20260619 06:48:29.000288 16969 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ecbe347d254beeb644d967a2fa5a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:28 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48716
I20260619 06:48:29.000291 16830 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ecbe347d254beeb644d967a2fa5a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:28 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48054
I20260619 06:48:29.000620 17205 tablet_replica.cc:330] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: stopping tablet replica
I20260619 06:48:29.000658 17206 tablet_replica.cc:330] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: stopping tablet replica
I20260619 06:48:29.000757 17205 raft_consensus.cc:2229] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:29.000793 17206 raft_consensus.cc:2229] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:29.000866 17205 raft_consensus.cc:2258] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:29.000934 17206 raft_consensus.cc:2258] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:29.001078 17204 tablet_replica.cc:330] T 56ecbe347d254beeb644d967a2fa5a1a P 9c76bf0c82024ff99ea0d14d9eaa4e0b: stopping tablet replica
I20260619 06:48:29.001106 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 16625
I20260619 06:48:29.002110 17206 ts_tablet_manager.cc:1905] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:29.002244 17205 ts_tablet_manager.cc:1905] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:29.004225 17206 ts_tablet_manager.cc:1918] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:48:29.004315 17206 log.cc:1196] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/56ecbe347d254beeb644d967a2fa5a1a
I20260619 06:48:29.004586 17206 ts_tablet_manager.cc:1939] T 56ecbe347d254beeb644d967a2fa5a1a P cd795e0e60e94a67838e492f67e31ef9: Deleting consensus metadata
I20260619 06:48:29.005177 16537 catalog_manager.cc:4841] TS cd795e0e60e94a67838e492f67e31ef9 (127.14.116.194:46333): tablet 56ecbe347d254beeb644d967a2fa5a1a (table test-table [id=d146c39ffd2d4f0688d83bc146345849]) successfully deleted
I20260619 06:48:29.005451 17205 ts_tablet_manager.cc:1918] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:48:29.005535 17205 log.cc:1196] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/56ecbe347d254beeb644d967a2fa5a1a
I20260619 06:48:29.005815 17205 ts_tablet_manager.cc:1939] T 56ecbe347d254beeb644d967a2fa5a1a P 445ec28deed947fea13fae313cc3ba0e: Deleting consensus metadata
I20260619 06:48:29.006500 16539 catalog_manager.cc:4841] TS 445ec28deed947fea13fae313cc3ba0e (127.14.116.195:34567): tablet 56ecbe347d254beeb644d967a2fa5a1a (table test-table [id=d146c39ffd2d4f0688d83bc146345849]) successfully deleted
W20260619 06:48:29.009467 16539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:39291 (error 108)
I20260619 06:48:29.009688 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 16765
I20260619 06:48:29.012365 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:29.011888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:39291 (local address 127.0.0.1:51688)
0619 06:48:29.012145 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:29.012235 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:39291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
W20260619 06:48:29.012518 16539 catalog_manager.cc:4584] TS 9c76bf0c82024ff99ea0d14d9eaa4e0b (127.14.116.193:39291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56ecbe347d254beeb644d967a2fa5a1a: Network error: Client connection negotiation failed: client connection to 127.14.116.193:39291: connect: Connection refused (error 111)
I20260619 06:48:29.016791 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 16905
I20260619 06:48:29.022979 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 16521
2026-06-19T06:48:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6176 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:29 dist-test-slave-9x3b krb5kdc[17211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:29 dist-test-slave-9x3b krb5kdc[17211](info): set up 2 sockets
Jun 19 06:48:29 dist-test-slave-9x3b krb5kdc[17211](info): commencing operation
krb5kdc: starting...
W20260619 06:48:30.107967 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 06:48:30.344810 14803 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.237s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 06:48:30.549057 14803 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.204s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 06:48:30.749177 14803 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.200s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:30 dist-test-slave-9x3b krb5kdc[17211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:30.874128 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:39419
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:39419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:30.980065 17224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:30.980305 17224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:30.980343 17224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:30.984256 17224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:30.984306 17224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:30.984326 17224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:30.984342 17224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:30.984364 17224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:30.989511 17224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:39419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:39419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:30.990708 17224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:30.991714 17224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:30.999783 17230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:30.999792 17232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:30.999792 17229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:31.000192 17224 server_base.cc:1034] running on GCE node
I20260619 06:48:31.000716 17224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:31.001744 17224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:31.002939 17224 hybrid_clock.cc:648] HybridClock initialized: now 1781851711002929 us; error 32 us; skew 500 ppm
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851711, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:31.005599 17224 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:31.006502 17224 webserver.cc:460] Webserver started at http://127.14.116.254:43221/ using document root <none> and password file <none>
I20260619 06:48:31.007081 17224 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:31.007175 17224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:31.007412 17224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:31.009470 17224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "e651560037ed4ab28ad84e6169aa4fa4"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "4c5ad1e260fac5b6981b878bc2bd62a6"
server_key_iv: "2d6e2dfb58fea12d2300a1ab2cfab985"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.010025 17224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "e651560037ed4ab28ad84e6169aa4fa4"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "4c5ad1e260fac5b6981b878bc2bd62a6"
server_key_iv: "2d6e2dfb58fea12d2300a1ab2cfab985"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.013615 17224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 06:48:31.015897 17239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.017052 17224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:48:31.017189 17224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "e651560037ed4ab28ad84e6169aa4fa4"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "4c5ad1e260fac5b6981b878bc2bd62a6"
server_key_iv: "2d6e2dfb58fea12d2300a1ab2cfab985"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.017295 17224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:31.029789 17224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:31.032537 17224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:31.032729 17224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:31.039479 17224 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:39419
I20260619 06:48:31.039486 17291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:39419 every 8 connection(s)
I20260619 06:48:31.040416 17224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:31.043006 17292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:31.048700 17292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4: Bootstrap starting.
I20260619 06:48:31.049062 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 17224
I20260619 06:48:31.049171 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:31.049422 14803 external_mini_cluster.cc:1428] Setting key 6670fbc84ad0ef9cb231ada1e897488c
I20260619 06:48:31.051115 17292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:31.051878 17292 log.cc:824] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:31.053923 17292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4: No bootstrap required, opened a new log
I20260619 06:48:31.058205 17292 raft_consensus.cc:348] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651560037ed4ab28ad84e6169aa4fa4" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39419 } }
I20260619 06:48:31.058400 17292 raft_consensus.cc:374] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:31.058458 17292 raft_consensus.cc:729] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e651560037ed4ab28ad84e6169aa4fa4, State: Initialized, Role: FOLLOWER
I20260619 06:48:31.058941 17292 consensus_queue.cc:260] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651560037ed4ab28ad84e6169aa4fa4" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39419 } }
I20260619 06:48:31.059060 17292 raft_consensus.cc:388] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:31.059142 17292 raft_consensus.cc:482] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:31.059240 17292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:31.060108 17292 raft_consensus.cc:504] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651560037ed4ab28ad84e6169aa4fa4" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39419 } }
I20260619 06:48:31.060434 17292 leader_election.cc:302] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e651560037ed4ab28ad84e6169aa4fa4; no voters:
I20260619 06:48:31.060712 17292 leader_election.cc:288] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:31.060832 17297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:31.061041 17297 raft_consensus.cc:686] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [term 1 LEADER]: Becoming Leader. State: Replica: e651560037ed4ab28ad84e6169aa4fa4, State: Running, Role: LEADER
I20260619 06:48:31.061430 17297 consensus_queue.cc:237] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651560037ed4ab28ad84e6169aa4fa4" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39419 } }
I20260619 06:48:31.061902 17292 sys_catalog.cc:564] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:31.063119 17298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e651560037ed4ab28ad84e6169aa4fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651560037ed4ab28ad84e6169aa4fa4" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39419 } } }
I20260619 06:48:31.063254 17298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:31.063488 17298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e651560037ed4ab28ad84e6169aa4fa4. Latest consensus state: current_term: 1 leader_uuid: "e651560037ed4ab28ad84e6169aa4fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651560037ed4ab28ad84e6169aa4fa4" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39419 } } }
I20260619 06:48:31.063577 17298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:31.063907 17307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:31.067305 17307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:31.070571 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:31.050696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.14.116.254:39419)
0619 06:48:31.051073 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:31.051097 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:31.051130 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:48:31.051855 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:31.051875 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:31.052273 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:31.052647 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:31.053531 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.054539 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.055723 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.056164 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.056275 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:31.059049 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:31.059057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:31.059072 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:31.059104 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:31.065485 (+ 6381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:31.066962 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:31.066982 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:31.066989 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:31.067127 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:31.067453 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:31.067456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:31.067458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:31.067973 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:31.068151 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:31.069436 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0619 06:48:31.069711 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":132,"threads_started":1}
I20260619 06:48:31.072569 17307 catalog_manager.cc:1347] Generated new cluster ID: 2f9bebe7458841f09197d7e5700df236
I20260619 06:48:31.072621 17307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:31.092981 17307 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:31.093870 17307 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:31.101164 17307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e651560037ed4ab28ad84e6169aa4fa4: Generated new TSK 0
I20260619 06:48:31.101858 17307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:31.128578 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39419
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:31.242246 17318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:31.242509 17318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:31.242576 17318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:31.246639 17318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:31.246719 17318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:31.246809 17318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:31.251899 17318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:31.252931 17318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:31.254082 17318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:31.263139 17326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:31.263139 17324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:31.263240 17323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:31.263305 17318 server_base.cc:1034] running on GCE node
I20260619 06:48:31.263856 17318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:31.264451 17318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:31.265627 17318 hybrid_clock.cc:648] HybridClock initialized: now 1781851711265609 us; error 36 us; skew 500 ppm
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851711, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:31.268211 17318 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:31.269042 17318 webserver.cc:460] Webserver started at http://127.14.116.193:46715/ using document root <none> and password file <none>
I20260619 06:48:31.269562 17318 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:31.269647 17318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:31.269874 17318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:31.271798 17318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "e7ccac8ae7ca47f6afc53be4264d0cc9"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "bf310f82efaf6ccc821fd6d2516d7a8e"
server_key_iv: "922a63784f06fdef3f40c2684405d2b7"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.272329 17318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "e7ccac8ae7ca47f6afc53be4264d0cc9"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "bf310f82efaf6ccc821fd6d2516d7a8e"
server_key_iv: "922a63784f06fdef3f40c2684405d2b7"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.275799 17318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:31.277995 17333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.279109 17318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:31.279234 17318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "e7ccac8ae7ca47f6afc53be4264d0cc9"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "bf310f82efaf6ccc821fd6d2516d7a8e"
server_key_iv: "922a63784f06fdef3f40c2684405d2b7"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.279342 17318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:31.291249 17318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:31.293891 17318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:31.294090 17318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:31.294647 17318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:31.295399 17318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:31.295466 17318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.295531 17318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:31.295578 17318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.304198 17446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:40399 every 8 connection(s)
I20260619 06:48:31.304204 17318 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:40399
I20260619 06:48:31.305058 17318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851711, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:31.314486 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 17318
I20260619 06:48:31.314615 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:31.314917 14803 external_mini_cluster.cc:1428] Setting key 951b25a8c58546e6a835fcf87b4750a4
I20260619 06:48:31.318360 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:31.306709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:33961 (local address 127.14.116.254:39419)
0619 06:48:31.306837 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:31.306841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:31.307641 (+ 800us) server_negotiation.cc:407] Connection header received
0619 06:48:31.308917 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:31.308920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:31.308975 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:31.309079 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:31.310175 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.310715 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.311860 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.312109 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.312220 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:31.314579 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:31.314585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:31.314588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:31.314613 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:31.316165 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:31.316740 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:31.316743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:31.316744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:31.316815 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:31.317187 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:31.317189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:31.317190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:31.317341 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:31.317415 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:31.318017 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 06:48:31.318153 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:48:31.319070 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:31.307068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:39419 (local address 127.14.116.193:33961)
0619 06:48:31.307483 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:31.307522 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:31.308705 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:31.309240 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:31.309246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:31.309584 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:31.310029 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:31.310041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.310854 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.310858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:31.311731 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:31.311739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.312253 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.312256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:31.312439 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:31.313051 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:31.313067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:31.314424 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:31.316301 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:31.316306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:31.316318 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:31.316640 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:31.316906 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:31.316911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:31.316916 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:31.317096 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:31.317439 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:31.317448 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:31.317747 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 06:48:31.317952 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 06:48:31.318262 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260619 06:48:31.320240 17447 heartbeater.cc:347] Connected to a master server at 127.14.116.254:39419
I20260619 06:48:31.320483 17447 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:31.321064 17447 heartbeater.cc:511] Master 127.14.116.254:39419 requested a full tablet report, sending...
I20260619 06:48:31.322741 17256 ts_manager.cc:194] Registered new tserver with Master: e7ccac8ae7ca47f6afc53be4264d0cc9 (127.14.116.193:40399)
I20260619 06:48:31.323891 17256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:33961
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:31.339663 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39419
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:31.444509 17452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:31.444779 17452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:31.444859 17452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:31.449278 17452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:31.449357 17452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:31.449458 17452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:31.454428 17452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:31.455350 17452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:31.456338 17452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:31.465550 17457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:31.465551 17460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:31.465749 17458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:31.465808 17452 server_base.cc:1034] running on GCE node
I20260619 06:48:31.466208 17452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:31.466816 17452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:31.467984 17452 hybrid_clock.cc:648] HybridClock initialized: now 1781851711467961 us; error 47 us; skew 500 ppm
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851711, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:31.470542 17452 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:31.471285 17452 webserver.cc:460] Webserver started at http://127.14.116.194:38517/ using document root <none> and password file <none>
I20260619 06:48:31.471833 17452 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:31.471892 17452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:31.472047 17452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:31.473876 17452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "cc0dc415832a40f7a442d2dc777ff1a4"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "f17a7c52920fbc6bf02e9b87d6722928"
server_key_iv: "dcb6425e72737b40b72f012cb0758cd0"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.474450 17452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "cc0dc415832a40f7a442d2dc777ff1a4"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "f17a7c52920fbc6bf02e9b87d6722928"
server_key_iv: "dcb6425e72737b40b72f012cb0758cd0"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.477898 17452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 06:48:31.480104 17467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.481204 17452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:31.481292 17452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "cc0dc415832a40f7a442d2dc777ff1a4"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "f17a7c52920fbc6bf02e9b87d6722928"
server_key_iv: "dcb6425e72737b40b72f012cb0758cd0"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.481365 17452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:31.496066 17452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:31.498646 17452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:31.498782 17452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:31.499248 17452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:31.500041 17452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:31.500087 17452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.500124 17452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:31.500146 17452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.508370 17452 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:42669
I20260619 06:48:31.508446 17580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:42669 every 8 connection(s)
I20260619 06:48:31.509210 17452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:31.514695 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 17452
I20260619 06:48:31.514780 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:31.515058 14803 external_mini_cluster.cc:1428] Setting key db505678b8259641da04b1adfc580302
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851711, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:31.522557 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:31.510844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:41881 (local address 127.14.116.254:39419)
0619 06:48:31.510979 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:31.510982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:31.511935 (+ 953us) server_negotiation.cc:407] Connection header received
0619 06:48:31.512788 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:31.512791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:31.512846 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:31.512957 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:31.514018 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.514543 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.516000 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.516262 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.516366 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:31.518846 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:31.518850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:31.518852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:31.518875 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:31.520498 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:31.521043 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:31.521049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:31.521051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:31.521133 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:31.521468 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:31.521473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:31.521474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:31.521655 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:31.521758 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:31.522271 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 06:48:31.522378 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":58}
I20260619 06:48:31.523257 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:31.511187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:39419 (local address 127.14.116.194:41881)
0619 06:48:31.511798 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:31.511834 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:31.512593 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:31.513118 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:31.513124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:31.513447 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:31.513872 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:31.513887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.514663 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.514667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:31.515873 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:31.515884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.516416 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.516421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:31.516601 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:31.517167 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:31.517181 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:31.518697 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:31.520602 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:31.520607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:31.520619 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:31.520949 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:31.521233 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:31.521237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:31.521240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:31.521373 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:31.521757 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:31.521763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:31.522046 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 06:48:31.522218 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 06:48:31.522509 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:48:31.524367 17581 heartbeater.cc:347] Connected to a master server at 127.14.116.254:39419
I20260619 06:48:31.524554 17581 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:31.525192 17581 heartbeater.cc:511] Master 127.14.116.254:39419 requested a full tablet report, sending...
I20260619 06:48:31.526188 17256 ts_manager.cc:194] Registered new tserver with Master: cc0dc415832a40f7a442d2dc777ff1a4 (127.14.116.194:42669)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:48:31.526867 17256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:41881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:31.540319 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39419
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:31.650910 17586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:31.651149 17586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:31.651190 17586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:31.655441 17586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:31.655499 17586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:31.655557 17586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:31.660743 17586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:31.661634 17586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:31.662696 17586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:31.671898 17591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:31.671898 17592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:31.671932 17586 server_base.cc:1034] running on GCE node
W20260619 06:48:31.671943 17594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:31.672564 17586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:31.673166 17586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:31.674355 17586 hybrid_clock.cc:648] HybridClock initialized: now 1781851711674325 us; error 43 us; skew 500 ppm
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851711, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:31.677052 17586 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:31.677794 17586 webserver.cc:460] Webserver started at http://127.14.116.195:32901/ using document root <none> and password file <none>
I20260619 06:48:31.678267 17586 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:31.678321 17586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:31.678481 17586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:31.680420 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "76403ad4a7a94aaba56abda9ab59461c"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "cd23a99ec2ff23e83f0c8369c6b0daa6"
server_key_iv: "50d787566534becfaf9eddb00a72af01"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.680934 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "76403ad4a7a94aaba56abda9ab59461c"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "cd23a99ec2ff23e83f0c8369c6b0daa6"
server_key_iv: "50d787566534becfaf9eddb00a72af01"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.684444 17586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:31.686738 17601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.687911 17586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:31.688000 17586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "76403ad4a7a94aaba56abda9ab59461c"
format_stamp: "Formatted at 2026-06-19 06:48:31 on dist-test-slave-9x3b"
server_key: "cd23a99ec2ff23e83f0c8369c6b0daa6"
server_key_iv: "50d787566534becfaf9eddb00a72af01"
server_key_version: "encryptionkey@0"
I20260619 06:48:31.688076 17586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:31.700516 17586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:31.703503 17586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:31.703704 17586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:31.704175 17586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:31.704874 17586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:31.704924 17586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.704962 17586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:31.704982 17586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:31.712958 17586 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:37407
I20260619 06:48:31.712988 17714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:37407 every 8 connection(s)
I20260619 06:48:31.713876 17586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:31.715514 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 17586
I20260619 06:48:31.715654 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:31.715947 14803 external_mini_cluster.cc:1428] Setting key e70983b4e8d509c21526a943ec9af08c
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851711, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:31.725991 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:31.715573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:47005 (local address 127.14.116.254:39419)
0619 06:48:31.715738 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:31.715743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:31.716825 (+ 1082us) server_negotiation.cc:407] Connection header received
0619 06:48:31.717657 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:31.717660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:31.717711 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:31.717792 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:31.718877 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.719379 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.720437 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.720707 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.720788 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:31.722992 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:31.722995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:31.722997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:31.723019 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:31.724450 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:31.724847 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:31.724849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:31.724851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:31.724916 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:31.725183 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:31.725185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:31.725186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:31.725336 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:31.725406 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:31.725790 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 06:48:31.725872 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:48:31.726716 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:31.715994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:39419 (local address 127.14.116.195:47005)
0619 06:48:31.716691 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:31.716741 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:31.717476 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:31.717969 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:31.717984 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:31.718308 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:31.718760 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:31.718773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.719478 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.719480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:31.720331 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:31.720338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:31.720848 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:31.720854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:31.721006 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:31.721472 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:31.721486 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:31.722846 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:31.724536 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:31.724540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:31.724550 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:31.724768 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:31.724995 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:31.724997 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:31.724998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:31.725111 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:31.725412 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:31.725416 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:31.725611 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 06:48:31.725750 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 06:48:31.725970 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":126,"threads_started":1}
I20260619 06:48:31.727752 17715 heartbeater.cc:347] Connected to a master server at 127.14.116.254:39419
I20260619 06:48:31.727977 17715 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:31.728485 17715 heartbeater.cc:511] Master 127.14.116.254:39419 requested a full tablet report, sending...
I20260619 06:48:31.729449 17256 ts_manager.cc:194] Registered new tserver with Master: 76403ad4a7a94aaba56abda9ab59461c (127.14.116.195:37407)
I20260619 06:48:31.729984 17256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:47005
I20260619 06:48:31.741129 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:48:31.753176 17295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:31.751983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.14.116.254:39419)
0619 06:48:31.752103 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:31.752106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:31.752242 (+ 136us) server_negotiation.cc:407] Connection header received
0619 06:48:31.752333 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:31.752335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:31.752374 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:31.752448 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:31.753078 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58870: BlockingRecv error: recv got EOF from 127.0.0.1:58870 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:48:31.754185 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 17318
I20260619 06:48:31.759253 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 17452
I20260619 06:48:31.764467 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 17586
I20260619 06:48:31.769273 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 17224
2026-06-19T06:48:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2744 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17727](info): set up 2 sockets
Jun 19 06:48:31 dist-test-slave-9x3b krb5kdc[17727](info): commencing operation
krb5kdc: starting...
W20260619 06:48:32.795521 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:32 dist-test-slave-9x3b krb5kdc[17727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:32.906498 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:42575
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:42575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:33.011458 17740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:33.011797 17740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:33.011883 17740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:33.015827 17740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:33.015913 17740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:33.015950 17740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:33.015986 17740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:33.016022 17740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:33.021059 17740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:42575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:42575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:33.021993 17740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:33.022995 17740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:33.030997 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:33.031036 17740 server_base.cc:1034] running on GCE node
W20260619 06:48:33.031023 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:33.031256 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:33.031807 17740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:33.032781 17740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:33.033934 17740 hybrid_clock.cc:648] HybridClock initialized: now 1781851713033918 us; error 27 us; skew 500 ppm
Jun 19 06:48:33 dist-test-slave-9x3b krb5kdc[17727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851713, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:33.041813 17740 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:33.042644 17740 webserver.cc:460] Webserver started at http://127.14.116.254:42173/ using document root <none> and password file <none>
I20260619 06:48:33.043125 17740 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:33.043180 17740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:33.043332 17740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:33.045233 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "399db35766ba4ce3b0c8bec6581b3a45"
format_stamp: "Formatted at 2026-06-19 06:48:33 on dist-test-slave-9x3b"
server_key: "a33c62c0e42413883799659cd2a152a3"
server_key_iv: "08e0d53c681881dff699dc1e130f4cf3"
server_key_version: "encryptionkey@0"
I20260619 06:48:33.045706 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "399db35766ba4ce3b0c8bec6581b3a45"
format_stamp: "Formatted at 2026-06-19 06:48:33 on dist-test-slave-9x3b"
server_key: "a33c62c0e42413883799659cd2a152a3"
server_key_iv: "08e0d53c681881dff699dc1e130f4cf3"
server_key_version: "encryptionkey@0"
I20260619 06:48:33.049182 17740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:33.051410 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:33.052673 17740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:33.052806 17740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "399db35766ba4ce3b0c8bec6581b3a45"
format_stamp: "Formatted at 2026-06-19 06:48:33 on dist-test-slave-9x3b"
server_key: "a33c62c0e42413883799659cd2a152a3"
server_key_iv: "08e0d53c681881dff699dc1e130f4cf3"
server_key_version: "encryptionkey@0"
I20260619 06:48:33.052917 17740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:33.085471 17740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:33.088268 17740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:33.088459 17740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:33.095116 17740 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:42575
I20260619 06:48:33.095357 17807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:42575 every 8 connection(s)
I20260619 06:48:33.096127 17740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:33.098670 17808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:33.101761 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 17740
I20260619 06:48:33.101881 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:33.102161 14803 external_mini_cluster.cc:1428] Setting key 891648eace0e39a21db34fb6f88b7889
I20260619 06:48:33.104357 17808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45: Bootstrap starting.
I20260619 06:48:33.106364 17808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:33.107059 17808 log.cc:824] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:33.108824 17808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45: No bootstrap required, opened a new log
Jun 19 06:48:33 dist-test-slave-9x3b krb5kdc[17727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:33.112882 17808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db35766ba4ce3b0c8bec6581b3a45" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42575 } }
I20260619 06:48:33.113148 17808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:33.113209 17808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399db35766ba4ce3b0c8bec6581b3a45, State: Initialized, Role: FOLLOWER
I20260619 06:48:33.113669 17808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db35766ba4ce3b0c8bec6581b3a45" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42575 } }
I20260619 06:48:33.113832 17808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:33.113901 17808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:33.113996 17808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:33.115031 17808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db35766ba4ce3b0c8bec6581b3a45" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42575 } }
I20260619 06:48:33.115376 17808 leader_election.cc:302] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 399db35766ba4ce3b0c8bec6581b3a45; no voters:
I20260619 06:48:33.115733 17808 leader_election.cc:288] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:33.115833 17813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:33.116025 17813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [term 1 LEADER]: Becoming Leader. State: Replica: 399db35766ba4ce3b0c8bec6581b3a45, State: Running, Role: LEADER
I20260619 06:48:33.116178 17811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:33.103431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.14.116.254:42575)
0619 06:48:33.103842 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:33.103856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:33.103896 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:48:33.104520 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:33.104536 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:33.104834 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:33.105169 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:33.105958 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:33.106972 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:33.107944 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:33.108396 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:33.108561 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:33.111214 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:33.111219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:33.111231 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:33.111257 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:33.112931 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:33.113416 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:33.113437 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:33.113445 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:33.113583 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:33.113907 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:33.113911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:33.113913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:33.114412 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:33.114577 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:33.114838 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 06:48:33.115114 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":136,"threads_started":1}
I20260619 06:48:33.116413 17813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db35766ba4ce3b0c8bec6581b3a45" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42575 } }
I20260619 06:48:33.116976 17808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:33.118057 17814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "399db35766ba4ce3b0c8bec6581b3a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db35766ba4ce3b0c8bec6581b3a45" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42575 } } }
I20260619 06:48:33.118176 17814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:33.118541 17822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:33.118999 17815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 399db35766ba4ce3b0c8bec6581b3a45. Latest consensus state: current_term: 1 leader_uuid: "399db35766ba4ce3b0c8bec6581b3a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db35766ba4ce3b0c8bec6581b3a45" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42575 } } }
I20260619 06:48:33.119091 17815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:33.119433 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:48:33.122057 17822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:33.128087 17822 catalog_manager.cc:1347] Generated new cluster ID: 0828569d1b3f4743adb12e8c145e89cf
I20260619 06:48:33.128187 17822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:33.135157 17822 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:33.136180 17822 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:33.141434 17811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:33.135820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.14.116.254:42575)
0619 06:48:33.135937 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:33.135942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:33.135957 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:48:33.135993 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:33.135996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:33.136213 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:33.136291 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:33.136939 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:33.137718 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:33.138940 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:33.139311 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:33.139381 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:33.140401 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:33.140405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:33.140408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:33.140943 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:33.140974 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:33.140992 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:33.141051 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:33.141222 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 06:48:33.141292 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:33.143678 17822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 399db35766ba4ce3b0c8bec6581b3a45: Generated new TSK 0
I20260619 06:48:33.144340 17822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:48:33.151340 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 17740
2026-06-19T06:48:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1379 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:33 dist-test-slave-9x3b krb5kdc[17841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:33 dist-test-slave-9x3b krb5kdc[17841](info): set up 2 sockets
Jun 19 06:48:33 dist-test-slave-9x3b krb5kdc[17841](info): commencing operation
krb5kdc: starting...
W20260619 06:48:34.187444 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:34 dist-test-slave-9x3b krb5kdc[17841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:34.298182 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:44847
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:44847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:34.405786 17854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:34.406067 17854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:34.406149 17854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:34.410113 17854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:34.410192 17854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:34.410229 17854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:34.410285 17854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:34.410334 17854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:34.415426 17854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:44847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:44847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:34.416426 17854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:34.417410 17854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:34.425454 17859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:34.425546 17854 server_base.cc:1034] running on GCE node
W20260619 06:48:34.425438 17860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:34.425438 17862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:34.426241 17854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:34.427222 17854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:34.428385 17854 hybrid_clock.cc:648] HybridClock initialized: now 1781851714428363 us; error 25 us; skew 500 ppm
Jun 19 06:48:34 dist-test-slave-9x3b krb5kdc[17841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851714, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:34.430886 17854 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:34.431757 17854 webserver.cc:460] Webserver started at http://127.14.116.254:44375/ using document root <none> and password file <none>
I20260619 06:48:34.432238 17854 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:34.432294 17854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:34.432466 17854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:34.434351 17854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "7eab1faa670f45a1bbe13fd8c73ff362"
format_stamp: "Formatted at 2026-06-19 06:48:34 on dist-test-slave-9x3b"
server_key: "d3da3f81986cf45492c542cebbefc923"
server_key_iv: "a7c0d57a0cef4e37078da90d61101562"
server_key_version: "encryptionkey@0"
I20260619 06:48:34.434846 17854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "7eab1faa670f45a1bbe13fd8c73ff362"
format_stamp: "Formatted at 2026-06-19 06:48:34 on dist-test-slave-9x3b"
server_key: "d3da3f81986cf45492c542cebbefc923"
server_key_iv: "a7c0d57a0cef4e37078da90d61101562"
server_key_version: "encryptionkey@0"
I20260619 06:48:34.438313 17854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:48:34.440572 17869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:34.441733 17854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:34.441854 17854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "7eab1faa670f45a1bbe13fd8c73ff362"
format_stamp: "Formatted at 2026-06-19 06:48:34 on dist-test-slave-9x3b"
server_key: "d3da3f81986cf45492c542cebbefc923"
server_key_iv: "a7c0d57a0cef4e37078da90d61101562"
server_key_version: "encryptionkey@0"
I20260619 06:48:34.441953 17854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:34.464056 17854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:34.466699 17854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:34.466914 17854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:34.473603 17854 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:44847
I20260619 06:48:34.473860 17921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:44847 every 8 connection(s)
I20260619 06:48:34.474552 17854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:34.477187 17922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:34.482355 17922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362: Bootstrap starting.
I20260619 06:48:34.483731 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 17854
I20260619 06:48:34.483850 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:34.484115 14803 external_mini_cluster.cc:1428] Setting key f9f015abb246de7eb8ef68e491c5e309
I20260619 06:48:34.485090 17922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:34.485880 17922 log.cc:824] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:34.487735 17922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362: No bootstrap required, opened a new log
I20260619 06:48:34.491142 17922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eab1faa670f45a1bbe13fd8c73ff362" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44847 } }
I20260619 06:48:34.491371 17922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:34.491451 17922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eab1faa670f45a1bbe13fd8c73ff362, State: Initialized, Role: FOLLOWER
I20260619 06:48:34.491909 17922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eab1faa670f45a1bbe13fd8c73ff362" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44847 } }
I20260619 06:48:34.492057 17922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:34.492137 17922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:34.492233 17922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 06:48:34 dist-test-slave-9x3b krb5kdc[17841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:34.493134 17922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eab1faa670f45a1bbe13fd8c73ff362" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44847 } }
I20260619 06:48:34.493443 17922 leader_election.cc:302] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eab1faa670f45a1bbe13fd8c73ff362; no voters:
I20260619 06:48:34.493789 17922 leader_election.cc:288] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:34.493852 17927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:34.494064 17927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [term 1 LEADER]: Becoming Leader. State: Replica: 7eab1faa670f45a1bbe13fd8c73ff362, State: Running, Role: LEADER
I20260619 06:48:34.494359 17927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eab1faa670f45a1bbe13fd8c73ff362" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44847 } }
I20260619 06:48:34.495050 17922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:34.495533 17929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eab1faa670f45a1bbe13fd8c73ff362. Latest consensus state: current_term: 1 leader_uuid: "7eab1faa670f45a1bbe13fd8c73ff362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eab1faa670f45a1bbe13fd8c73ff362" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44847 } } }
I20260619 06:48:34.496294 17929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:34.496176 17928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eab1faa670f45a1bbe13fd8c73ff362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eab1faa670f45a1bbe13fd8c73ff362" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44847 } } }
I20260619 06:48:34.496626 17928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:34.497241 17936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:34.498615 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:34.485220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.14.116.254:44847)
0619 06:48:34.485647 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:34.485664 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:34.485698 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:48:34.486473 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:34.486497 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:34.486846 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:34.487168 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:34.488016 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.489010 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.489861 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.490326 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.490533 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:34.493175 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:34.493180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:34.493192 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:34.493215 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:34.495246 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:34.495787 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:34.495793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:34.495798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:34.495962 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:34.496313 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:34.496316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:34.496318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:34.496828 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:34.497040 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:34.497472 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 06:48:34.497776 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20260619 06:48:34.500355 17936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:34.505733 17936 catalog_manager.cc:1347] Generated new cluster ID: 25c3fdf1fa814cb7b2b909c7b3493045
I20260619 06:48:34.505808 17936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:34.527700 17936 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:34.528594 17936 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:34.532513 17936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eab1faa670f45a1bbe13fd8c73ff362: Generated new TSK 0
I20260619 06:48:34.533161 17936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:34.566289 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44847
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:34.699586 17948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:34.699951 17948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:34.700008 17948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:34.703907 17948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:34.703953 17948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:34.704007 17948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:34.708995 17948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:34.709867 17948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:34.710759 17948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:34.719870 17954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:34.720068 17948 server_base.cc:1034] running on GCE node
W20260619 06:48:34.720240 17953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:34.720299 17956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:34.720681 17948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:34.721252 17948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:34.722435 17948 hybrid_clock.cc:648] HybridClock initialized: now 1781851714722413 us; error 45 us; skew 500 ppm
Jun 19 06:48:34 dist-test-slave-9x3b krb5kdc[17841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851714, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:34.725055 17948 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:34.726119 17948 webserver.cc:460] Webserver started at http://127.14.116.193:35507/ using document root <none> and password file <none>
I20260619 06:48:34.726681 17948 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:34.726780 17948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:34.727005 17948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:34.729066 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "c771a2373c964e308c4234a900fb41a2"
format_stamp: "Formatted at 2026-06-19 06:48:34 on dist-test-slave-9x3b"
server_key: "49f3081250fd6ba4280b43e5c56b8760"
server_key_iv: "b1e27dd2840f853dcbbb38339e79669c"
server_key_version: "encryptionkey@0"
I20260619 06:48:34.729625 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "c771a2373c964e308c4234a900fb41a2"
format_stamp: "Formatted at 2026-06-19 06:48:34 on dist-test-slave-9x3b"
server_key: "49f3081250fd6ba4280b43e5c56b8760"
server_key_iv: "b1e27dd2840f853dcbbb38339e79669c"
server_key_version: "encryptionkey@0"
I20260619 06:48:34.733110 17948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:34.735390 17963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:34.736608 17948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:34.736749 17948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "c771a2373c964e308c4234a900fb41a2"
format_stamp: "Formatted at 2026-06-19 06:48:34 on dist-test-slave-9x3b"
server_key: "49f3081250fd6ba4280b43e5c56b8760"
server_key_iv: "b1e27dd2840f853dcbbb38339e79669c"
server_key_version: "encryptionkey@0"
I20260619 06:48:34.736856 17948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:34.757862 17948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:34.760668 17948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:34.760860 17948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:34.761422 17948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:34.762188 17948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:34.762259 17948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:34.762327 17948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:34.762374 17948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:34.771013 17948 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:42827
I20260619 06:48:34.771039 18076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:42827 every 8 connection(s)
I20260619 06:48:34.771886 17948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:34.772490 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 17948
I20260619 06:48:34.772624 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:34.772925 14803 external_mini_cluster.cc:1428] Setting key 63d922387ad7418e022169cfef41ad4a
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Jun 19 06:48:34 dist-test-slave-9x3b krb5kdc[17841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851714, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
I20260619 06:48:34.786226 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:34.774426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:58531 (local address 127.14.116.254:44847)
0619 06:48:34.774547 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:34.774551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:34.775540 (+ 989us) server_negotiation.cc:407] Connection header received
0619 06:48:34.776995 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:34.776998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:34.777040 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:34.777136 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:34.778255 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.778772 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.779943 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.780170 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.780278 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:34.782785 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:34.782788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:34.782790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:34.782813 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:34.784153 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:34.784721 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:34.784728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:34.784732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:34.784821 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:34.785130 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:34.785136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:34.785141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:34.785396 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:34.785500 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:34.785939 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 06:48:34.786059 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:48:34.787056 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:34.774870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44847 (local address 127.14.116.193:58531)
0619 06:48:34.775372 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:34.775427 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:34.776777 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:34.777312 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:34.777319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:34.777661 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:34.778118 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:34.778130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.778893 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.778897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:34.779826 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:34.779835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.780310 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.780314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:34.780484 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:34.781074 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:34.781089 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:34.782641 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:34.784257 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:34.784266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:34.784284 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:34.784625 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:34.784912 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:34.784914 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:34.784916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:34.785040 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:34.785495 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:34.785499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:34.785712 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 06:48:34.785894 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 06:48:34.786201 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260619 06:48:34.788376 18077 heartbeater.cc:347] Connected to a master server at 127.14.116.254:44847
I20260619 06:48:34.788620 18077 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:34.789088 18077 heartbeater.cc:511] Master 127.14.116.254:44847 requested a full tablet report, sending...
I20260619 06:48:34.790503 17886 ts_manager.cc:194] Registered new tserver with Master: c771a2373c964e308c4234a900fb41a2 (127.14.116.193:42827)
I20260619 06:48:34.791584 17886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:58531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:34.798789 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44847
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:34.905383 18082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:34.905637 18082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:34.905702 18082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:34.909632 18082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:34.909711 18082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:34.909801 18082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:34.914853 18082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:34.915905 18082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:34.916890 18082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:34.926038 18087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:34.926121 18090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:34.926044 18088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:34.926230 18082 server_base.cc:1034] running on GCE node
I20260619 06:48:34.926679 18082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:34.927269 18082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:34.928447 18082 hybrid_clock.cc:648] HybridClock initialized: now 1781851714928432 us; error 27 us; skew 500 ppm
Jun 19 06:48:34 dist-test-slave-9x3b krb5kdc[17841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851714, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:34.930979 18082 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:34.931859 18082 webserver.cc:460] Webserver started at http://127.14.116.194:41415/ using document root <none> and password file <none>
I20260619 06:48:34.932387 18082 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:34.932474 18082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:34.932693 18082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:34.934638 18082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "0812fc95b63946f8bd540b0cdc2fe6fe"
format_stamp: "Formatted at 2026-06-19 06:48:34 on dist-test-slave-9x3b"
server_key: "d1e63cc7c747dc88822141cd2f5fffc8"
server_key_iv: "806911e0e675d7e145f0a54f65543eda"
server_key_version: "encryptionkey@0"
I20260619 06:48:34.935170 18082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "0812fc95b63946f8bd540b0cdc2fe6fe"
format_stamp: "Formatted at 2026-06-19 06:48:34 on dist-test-slave-9x3b"
server_key: "d1e63cc7c747dc88822141cd2f5fffc8"
server_key_iv: "806911e0e675d7e145f0a54f65543eda"
server_key_version: "encryptionkey@0"
I20260619 06:48:34.938714 18082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 06:48:34.941320 18097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:34.942446 18082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:34.942572 18082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "0812fc95b63946f8bd540b0cdc2fe6fe"
format_stamp: "Formatted at 2026-06-19 06:48:34 on dist-test-slave-9x3b"
server_key: "d1e63cc7c747dc88822141cd2f5fffc8"
server_key_iv: "806911e0e675d7e145f0a54f65543eda"
server_key_version: "encryptionkey@0"
I20260619 06:48:34.942673 18082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:34.954061 18082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:34.956619 18082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:34.956800 18082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:34.957317 18082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:34.958062 18082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:34.958132 18082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:34.958200 18082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:34.958243 18082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:34.966277 18082 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:34219
I20260619 06:48:34.966300 18210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:34219 every 8 connection(s)
I20260619 06:48:34.967072 18082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:34.974689 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 18082
I20260619 06:48:34.974831 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:34.975106 14803 external_mini_cluster.cc:1428] Setting key fbcc16eded6df6a2a80b6be70575d5e2
Jun 19 06:48:34 dist-test-slave-9x3b krb5kdc[17841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851714, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:34.979830 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:34.968760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:33329 (local address 127.14.116.254:44847)
0619 06:48:34.968882 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:34.968886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:34.969665 (+ 779us) server_negotiation.cc:407] Connection header received
0619 06:48:34.970903 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:34.970906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:34.970967 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:34.971038 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:34.972159 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.972660 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.973710 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.973914 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.973985 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:34.976570 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:34.976573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:34.976576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:34.976600 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:34.977890 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:34.978361 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:34.978366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:34.978368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:34.978452 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:34.978771 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:34.978776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:34.978778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:34.978967 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:34.979073 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:34.979524 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 06:48:34.979666 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:34.980631 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:34.969115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44847 (local address 127.14.116.194:33329)
0619 06:48:34.969522 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:34.969560 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:34.970718 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:34.971202 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:34.971209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:34.971539 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:34.972042 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:34.972054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.972756 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.972759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:34.973615 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:34.973623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:34.974029 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:34.974031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:34.974152 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:34.974738 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:34.974754 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:34.976421 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:34.978002 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:34.978008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:34.978021 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:34.978261 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:34.978548 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:34.978551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:34.978553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:34.978682 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:34.979070 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:34.979076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:34.979300 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 06:48:34.979482 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 06:48:34.979798 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260619 06:48:34.981870 18211 heartbeater.cc:347] Connected to a master server at 127.14.116.254:44847
I20260619 06:48:34.982095 18211 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:34.982538 18211 heartbeater.cc:511] Master 127.14.116.254:44847 requested a full tablet report, sending...
I20260619 06:48:34.983551 17886 ts_manager.cc:194] Registered new tserver with Master: 0812fc95b63946f8bd540b0cdc2fe6fe (127.14.116.194:34219)
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:48:34.984360 17886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:33329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:34.999872 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44847
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:35.108865 18216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:35.109117 18216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:35.109159 18216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:35.113093 18216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:35.113142 18216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:35.113196 18216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:35.118269 18216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:35.119203 18216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:35.120200 18216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:48:35.129525 18216 server_base.cc:1034] running on GCE node
W20260619 06:48:35.129489 18222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:35.129491 18221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:35.129491 18224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:35.130074 18216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:35.130646 18216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:35.131793 18216 hybrid_clock.cc:648] HybridClock initialized: now 1781851715131759 us; error 45 us; skew 500 ppm
Jun 19 06:48:35 dist-test-slave-9x3b krb5kdc[17841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851715, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:35.134325 18216 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:35.135061 18216 webserver.cc:460] Webserver started at http://127.14.116.195:39393/ using document root <none> and password file <none>
I20260619 06:48:35.135548 18216 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:35.135602 18216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:35.135836 18216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:35.137676 18216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "539442c9cdd944fe8e11ff35b331246e"
format_stamp: "Formatted at 2026-06-19 06:48:35 on dist-test-slave-9x3b"
server_key: "68d8450147659afcb6053ef84d540438"
server_key_iv: "63a298b1d943dbdec84257b8dac79f2c"
server_key_version: "encryptionkey@0"
I20260619 06:48:35.138147 18216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "539442c9cdd944fe8e11ff35b331246e"
format_stamp: "Formatted at 2026-06-19 06:48:35 on dist-test-slave-9x3b"
server_key: "68d8450147659afcb6053ef84d540438"
server_key_iv: "63a298b1d943dbdec84257b8dac79f2c"
server_key_version: "encryptionkey@0"
I20260619 06:48:35.141716 18216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:35.143941 18231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:35.145160 18216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:48:35.145305 18216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "539442c9cdd944fe8e11ff35b331246e"
format_stamp: "Formatted at 2026-06-19 06:48:35 on dist-test-slave-9x3b"
server_key: "68d8450147659afcb6053ef84d540438"
server_key_iv: "63a298b1d943dbdec84257b8dac79f2c"
server_key_version: "encryptionkey@0"
I20260619 06:48:35.145411 18216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:35.159444 18216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:35.162104 18216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:35.162293 18216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:35.162842 18216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:35.163650 18216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:35.163724 18216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:35.163789 18216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:35.163837 18216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:35.172389 18216 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:36537
I20260619 06:48:35.172416 18344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:36537 every 8 connection(s)
I20260619 06:48:35.173216 18216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:35.175383 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 18216
I20260619 06:48:35.175511 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:35.175843 14803 external_mini_cluster.cc:1428] Setting key 42f26f2b6d4fb0d69c2f14d2677e2e12
Jun 19 06:48:35 dist-test-slave-9x3b krb5kdc[17841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851715, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:35.186159 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:35.174844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:37879 (local address 127.14.116.254:44847)
0619 06:48:35.174970 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:35.174974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:35.175883 (+ 909us) server_negotiation.cc:407] Connection header received
0619 06:48:35.177446 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:35.177449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:35.177504 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:35.177603 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:35.178688 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:35.179211 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:35.180369 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:35.180591 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:35.180697 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:35.182902 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:35.182907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:35.182914 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:35.182938 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:35.184251 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:35.184726 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:35.184732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:35.184734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:35.184814 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:35.185138 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:35.185143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:35.185145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:35.185319 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:35.185423 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:35.185884 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 06:48:35.185997 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:48:35.186885 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:35.175184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44847 (local address 127.14.116.195:37879)
0619 06:48:35.175744 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:35.175784 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:35.177248 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:35.177775 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:35.177782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:35.178113 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:35.178558 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:35.178570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:35.179326 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:35.179330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:35.180252 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:35.180261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:35.180728 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:35.180732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:35.180893 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:35.181407 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:35.181422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:35.182751 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:35.184362 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:35.184369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:35.184382 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:35.184628 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:35.184909 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:35.184912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:35.184914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:35.185047 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:35.185420 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:35.185427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:35.185659 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 06:48:35.185834 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 06:48:35.186140 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":150,"threads_started":1}
I20260619 06:48:35.187943 18345 heartbeater.cc:347] Connected to a master server at 127.14.116.254:44847
I20260619 06:48:35.188213 18345 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:35.188752 18345 heartbeater.cc:511] Master 127.14.116.254:44847 requested a full tablet report, sending...
I20260619 06:48:35.189792 17886 ts_manager.cc:194] Registered new tserver with Master: 539442c9cdd944fe8e11ff35b331246e (127.14.116.195:36537)
I20260619 06:48:35.190326 17886 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:37879
I20260619 06:48:35.201013 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:48:35 dist-test-slave-9x3b krb5kdc[17841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851715, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 06:48:35 dist-test-slave-9x3b krb5kdc[17841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851715, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:35.225556 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:35.217072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.14.116.254:44847)
0619 06:48:35.217242 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:35.217247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:35.217394 (+ 147us) server_negotiation.cc:407] Connection header received
0619 06:48:35.217550 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:35.217553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:35.217604 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:35.217693 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:35.218370 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:35.218906 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:35.219795 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:35.220015 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:35.220123 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:35.222435 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:35.222438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:35.222441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:35.222463 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:35.223929 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:35.224333 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:35.224339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:35.224340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:35.224425 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:35.224697 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:35.224700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:35.224702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:35.224918 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:35.225058 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:35.225275 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:48:35.225394 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 06:48:35.225936 17886 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54494
I20260619 06:48:35.226780 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 17948
I20260619 06:48:35.231668 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 18082
I20260619 06:48:35.236696 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 18216
I20260619 06:48:35.241513 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 17854
2026-06-19T06:48:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2092 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:35 dist-test-slave-9x3b krb5kdc[18357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:35 dist-test-slave-9x3b krb5kdc[18357](info): set up 2 sockets
Jun 19 06:48:35 dist-test-slave-9x3b krb5kdc[18357](info): commencing operation
krb5kdc: starting...
W20260619 06:48:36.263823 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:36 dist-test-slave-9x3b krb5kdc[18357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:36.375761 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:43591
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:36.479956 18370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:36.480208 18370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:36.480283 18370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:36.484292 18370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:36.484341 18370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:36.484360 18370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:36.484412 18370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:36.484448 18370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:36.489658 18370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:36.490659 18370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:36.491707 18370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:36.499773 18378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:36.499778 18375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:36.499789 18376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:36.500113 18370 server_base.cc:1034] running on GCE node
I20260619 06:48:36.500689 18370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:36.501667 18370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:36.502818 18370 hybrid_clock.cc:648] HybridClock initialized: now 1781851716502789 us; error 41 us; skew 500 ppm
Jun 19 06:48:36 dist-test-slave-9x3b krb5kdc[18357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851716, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:36.505563 18370 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:36.506399 18370 webserver.cc:460] Webserver started at http://127.14.116.254:34955/ using document root <none> and password file <none>
I20260619 06:48:36.506949 18370 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:36.507040 18370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:36.507263 18370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:36.509284 18370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "93b592b4646543769d3135cbbc964864"
format_stamp: "Formatted at 2026-06-19 06:48:36 on dist-test-slave-9x3b"
server_key: "7d667d9c25b06322d734692a1b3e47bb"
server_key_iv: "555af574a4b47345f24f106c3b4394d6"
server_key_version: "encryptionkey@0"
I20260619 06:48:36.509828 18370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "93b592b4646543769d3135cbbc964864"
format_stamp: "Formatted at 2026-06-19 06:48:36 on dist-test-slave-9x3b"
server_key: "7d667d9c25b06322d734692a1b3e47bb"
server_key_iv: "555af574a4b47345f24f106c3b4394d6"
server_key_version: "encryptionkey@0"
I20260619 06:48:36.513444 18370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 06:48:36.515730 18385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:36.516901 18370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:36.517042 18370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "93b592b4646543769d3135cbbc964864"
format_stamp: "Formatted at 2026-06-19 06:48:36 on dist-test-slave-9x3b"
server_key: "7d667d9c25b06322d734692a1b3e47bb"
server_key_iv: "555af574a4b47345f24f106c3b4394d6"
server_key_version: "encryptionkey@0"
I20260619 06:48:36.517145 18370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:36.538782 18370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:36.541555 18370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:36.541749 18370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:36.548471 18370 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:43591
I20260619 06:48:36.548498 18437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:43591 every 8 connection(s)
I20260619 06:48:36.549343 18370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:36.551069 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 18370
I20260619 06:48:36.551195 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:36.551469 14803 external_mini_cluster.cc:1428] Setting key 574c57b60f9a4908fd1e430031146d91
I20260619 06:48:36.552134 18438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:36.558713 18438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864: Bootstrap starting.
Jun 19 06:48:36 dist-test-slave-9x3b krb5kdc[18357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:36.561064 18438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:36.561926 18438 log.cc:824] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:36.564108 18438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864: No bootstrap required, opened a new log
I20260619 06:48:36.566073 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:36.552868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.14.116.254:43591)
0619 06:48:36.553264 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:36.553283 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:36.553323 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:48:36.554113 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:36.554135 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:36.554547 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:36.554906 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:36.555528 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:36.557025 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:36.557930 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:36.558575 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:36.558742 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:36.561095 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:36.561102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:36.561120 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:36.561148 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:36.562993 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:36.563384 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:36.563389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:36.563394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:36.563518 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:36.563817 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:36.563821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:36.563823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:36.564293 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:36.564477 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:36.564668 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 06:48:36.564945 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260619 06:48:36.567790 18438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b592b4646543769d3135cbbc964864" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43591 } }
I20260619 06:48:36.568030 18438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:36.568101 18438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b592b4646543769d3135cbbc964864, State: Initialized, Role: FOLLOWER
I20260619 06:48:36.568506 18438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b592b4646543769d3135cbbc964864" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43591 } }
I20260619 06:48:36.568619 18438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:36.568660 18438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:36.568727 18438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:36.569491 18438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b592b4646543769d3135cbbc964864" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43591 } }
I20260619 06:48:36.569759 18438 leader_election.cc:302] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93b592b4646543769d3135cbbc964864; no voters:
I20260619 06:48:36.569988 18438 leader_election.cc:288] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:36.570109 18443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:36.570349 18443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [term 1 LEADER]: Becoming Leader. State: Replica: 93b592b4646543769d3135cbbc964864, State: Running, Role: LEADER
I20260619 06:48:36.570634 18443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b592b4646543769d3135cbbc964864" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43591 } }
I20260619 06:48:36.571161 18438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:36.572333 18444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93b592b4646543769d3135cbbc964864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b592b4646543769d3135cbbc964864" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43591 } } }
I20260619 06:48:36.572371 18445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93b592b4646543769d3135cbbc964864. Latest consensus state: current_term: 1 leader_uuid: "93b592b4646543769d3135cbbc964864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b592b4646543769d3135cbbc964864" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43591 } } }
I20260619 06:48:36.572477 18445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:36.572477 18444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:36.572887 18452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:36.576161 18452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:36.580988 18452 catalog_manager.cc:1347] Generated new cluster ID: b43caa7d6c264a899d505320a5961e65
I20260619 06:48:36.581064 18452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:36.589897 18452 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:36.591032 18452 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:36.603500 18452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93b592b4646543769d3135cbbc964864: Generated new TSK 0
I20260619 06:48:36.604205 18452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:36.631728 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43591
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:36.738829 18464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:36.739070 18464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:36.739111 18464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:36.743048 18464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:36.743098 18464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:36.743194 18464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:36.748286 18464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:36.749239 18464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:36.750209 18464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:36.759557 18472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:36.759557 18470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:36.759585 18469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:36.759723 18464 server_base.cc:1034] running on GCE node
I20260619 06:48:36.760076 18464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:36.760699 18464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:36.761886 18464 hybrid_clock.cc:648] HybridClock initialized: now 1781851716761859 us; error 36 us; skew 500 ppm
Jun 19 06:48:36 dist-test-slave-9x3b krb5kdc[18357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851716, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:36.764470 18464 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:36.765316 18464 webserver.cc:460] Webserver started at http://127.14.116.193:42241/ using document root <none> and password file <none>
I20260619 06:48:36.765851 18464 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:36.765945 18464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:36.766163 18464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:36.768187 18464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "8a74513538a94f63b45441f952f7ac09"
format_stamp: "Formatted at 2026-06-19 06:48:36 on dist-test-slave-9x3b"
server_key: "4aded2fd486ef1bd972de05dc7bbd126"
server_key_iv: "12081d6d0bad4f0ffadbdeab50d2e75c"
server_key_version: "encryptionkey@0"
I20260619 06:48:36.768719 18464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "8a74513538a94f63b45441f952f7ac09"
format_stamp: "Formatted at 2026-06-19 06:48:36 on dist-test-slave-9x3b"
server_key: "4aded2fd486ef1bd972de05dc7bbd126"
server_key_iv: "12081d6d0bad4f0ffadbdeab50d2e75c"
server_key_version: "encryptionkey@0"
I20260619 06:48:36.772334 18464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:36.774722 18479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:36.775928 18464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:36.776049 18464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "8a74513538a94f63b45441f952f7ac09"
format_stamp: "Formatted at 2026-06-19 06:48:36 on dist-test-slave-9x3b"
server_key: "4aded2fd486ef1bd972de05dc7bbd126"
server_key_iv: "12081d6d0bad4f0ffadbdeab50d2e75c"
server_key_version: "encryptionkey@0"
I20260619 06:48:36.776150 18464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:36.813230 18464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:36.816077 18464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:36.816270 18464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:36.816838 18464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:36.817631 18464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:36.817698 18464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:36.817768 18464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:36.817821 18464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:36.826514 18464 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:40235
I20260619 06:48:36.826542 18592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:40235 every 8 connection(s)
I20260619 06:48:36.827343 18464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
Jun 19 06:48:36 dist-test-slave-9x3b krb5kdc[18357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851716, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:36.837594 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 18464
I20260619 06:48:36.837750 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:36.838068 14803 external_mini_cluster.cc:1428] Setting key 60f4f8d76244db97bd07ca77ed91fb0c
I20260619 06:48:36.841207 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:36.829032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:43119 (local address 127.14.116.254:43591)
0619 06:48:36.829162 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:36.829166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:36.829994 (+ 828us) server_negotiation.cc:407] Connection header received
0619 06:48:36.831288 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:36.831290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:36.831337 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:36.831414 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:36.832961 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:36.833605 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:36.834743 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:36.834960 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:36.835034 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:36.837411 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:36.837414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:36.837416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:36.837438 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:36.839001 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:36.839596 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:36.839601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:36.839602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:36.839712 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:36.840026 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:36.840029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:36.840031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:36.840270 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:36.840385 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:36.840829 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 06:48:36.840967 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:48:36.842108 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:36.829425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:43591 (local address 127.14.116.193:43119)
0619 06:48:36.829837 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:36.829878 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:36.831096 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:36.831631 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:36.831638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:36.831997 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:36.832753 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:36.832775 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:36.833733 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:36.833736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:36.834639 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:36.834648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:36.835086 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:36.835088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:36.835226 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:36.835859 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:36.835874 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:36.837259 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:36.839110 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:36.839115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:36.839129 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:36.839488 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:36.839800 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:36.839802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:36.839803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:36.839931 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:36.840372 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:36.840376 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:36.840590 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 06:48:36.840753 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 06:48:36.840995 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260619 06:48:36.843364 18593 heartbeater.cc:347] Connected to a master server at 127.14.116.254:43591
I20260619 06:48:36.843636 18593 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:36.844231 18593 heartbeater.cc:511] Master 127.14.116.254:43591 requested a full tablet report, sending...
I20260619 06:48:36.845938 18402 ts_manager.cc:194] Registered new tserver with Master: 8a74513538a94f63b45441f952f7ac09 (127.14.116.193:40235)
I20260619 06:48:36.847584 18402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:43119
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:36.864492 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43591
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:36.969918 18598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:36.970144 18598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:36.970185 18598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:36.974258 18598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:36.974315 18598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:36.974370 18598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:36.979383 18598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:36.980417 18598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:36.981304 18598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:36.990206 18603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:36.990206 18604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:36.990439 18606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:36.990530 18598 server_base.cc:1034] running on GCE node
I20260619 06:48:36.990949 18598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:36.991518 18598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:36.992671 18598 hybrid_clock.cc:648] HybridClock initialized: now 1781851716992642 us; error 42 us; skew 500 ppm
Jun 19 06:48:36 dist-test-slave-9x3b krb5kdc[18357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851716, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:36.995265 18598 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:36.996097 18598 webserver.cc:460] Webserver started at http://127.14.116.194:44715/ using document root <none> and password file <none>
I20260619 06:48:36.996615 18598 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:36.996672 18598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:36.996896 18598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:36.998793 18598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "18a4fb1d60ee42a9bf41d1c97f772b79"
format_stamp: "Formatted at 2026-06-19 06:48:36 on dist-test-slave-9x3b"
server_key: "356b75929b0555b86df88013622a892f"
server_key_iv: "43120a2c1148aa9fe3854743b0f68ae7"
server_key_version: "encryptionkey@0"
I20260619 06:48:36.999333 18598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "18a4fb1d60ee42a9bf41d1c97f772b79"
format_stamp: "Formatted at 2026-06-19 06:48:36 on dist-test-slave-9x3b"
server_key: "356b75929b0555b86df88013622a892f"
server_key_iv: "43120a2c1148aa9fe3854743b0f68ae7"
server_key_version: "encryptionkey@0"
I20260619 06:48:37.002717 18598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:48:37.005163 18613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:37.006294 18598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:37.006407 18598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "18a4fb1d60ee42a9bf41d1c97f772b79"
format_stamp: "Formatted at 2026-06-19 06:48:36 on dist-test-slave-9x3b"
server_key: "356b75929b0555b86df88013622a892f"
server_key_iv: "43120a2c1148aa9fe3854743b0f68ae7"
server_key_version: "encryptionkey@0"
I20260619 06:48:37.006500 18598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:37.023332 18598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:37.025815 18598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:37.026007 18598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:37.026537 18598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:37.027282 18598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:37.027326 18598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:37.027392 18598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:37.027426 18598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:37.036125 18598 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:46435
I20260619 06:48:37.036146 18726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:46435 every 8 connection(s)
I20260619 06:48:37.037055 18598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:37.040009 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 18598
I20260619 06:48:37.040140 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:37.040508 14803 external_mini_cluster.cc:1428] Setting key 1f415fb8b12f7f9247d2aa394800a305
Jun 19 06:48:37 dist-test-slave-9x3b krb5kdc[18357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851716, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:48:37.051247 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:37.038707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:52353 (local address 127.14.116.254:43591)
0619 06:48:37.038848 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:37.038852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:37.039678 (+ 826us) server_negotiation.cc:407] Connection header received
0619 06:48:37.041458 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:37.041461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:37.041504 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:37.041586 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:37.043087 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.043591 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.044736 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.044961 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.045073 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:37.047866 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:37.047870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:37.047871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:37.047893 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:37.049292 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:37.049877 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:37.049880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:37.049881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:37.049949 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:37.050318 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:37.050320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:37.050321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:37.050486 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:37.050559 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:37.051003 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 06:48:37.051133 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:48:37.052139 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:37.039115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:43591 (local address 127.14.116.194:52353)
0619 06:48:37.039520 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:37.039560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:37.041243 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:37.041763 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:37.041775 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:37.042253 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:37.042929 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:37.042950 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.043763 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.043766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:37.044614 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:37.044622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.045102 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.045106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:37.045270 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:37.045873 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:37.045888 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:37.047709 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:37.049417 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:37.049424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:37.049441 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:37.049780 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:37.050038 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:37.050042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:37.050047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:37.050224 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:37.050578 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:37.050582 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:37.050788 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 06:48:37.050993 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 06:48:37.051237 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:48:37.053365 18727 heartbeater.cc:347] Connected to a master server at 127.14.116.254:43591
I20260619 06:48:37.053594 18727 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:37.054056 18727 heartbeater.cc:511] Master 127.14.116.254:43591 requested a full tablet report, sending...
I20260619 06:48:37.055091 18402 ts_manager.cc:194] Registered new tserver with Master: 18a4fb1d60ee42a9bf41d1c97f772b79 (127.14.116.194:46435)
I20260619 06:48:37.055701 18402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:52353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:37.066669 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43591
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:37.174556 18732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:37.174793 18732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:37.174834 18732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:37.178682 18732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:37.178727 18732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:37.178782 18732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:37.183830 18732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:37.184722 18732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:37.185626 18732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:37.194350 18740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:37.194412 18738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:37.194567 18732 server_base.cc:1034] running on GCE node
W20260619 06:48:37.194361 18737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:37.195055 18732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:37.195688 18732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:37.196856 18732 hybrid_clock.cc:648] HybridClock initialized: now 1781851717196834 us; error 41 us; skew 500 ppm
Jun 19 06:48:37 dist-test-slave-9x3b krb5kdc[18357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851717, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:37.199298 18732 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:37.200130 18732 webserver.cc:460] Webserver started at http://127.14.116.195:35899/ using document root <none> and password file <none>
I20260619 06:48:37.200650 18732 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:37.200735 18732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:37.200955 18732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:37.202859 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "bdd593b047584f8a801750bcdcdd23f2"
format_stamp: "Formatted at 2026-06-19 06:48:37 on dist-test-slave-9x3b"
server_key: "addf6b8451d492fd4ddd342bed132f9a"
server_key_iv: "2583d9a00a06b889b7b0951f19ef88b8"
server_key_version: "encryptionkey@0"
I20260619 06:48:37.203418 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "bdd593b047584f8a801750bcdcdd23f2"
format_stamp: "Formatted at 2026-06-19 06:48:37 on dist-test-slave-9x3b"
server_key: "addf6b8451d492fd4ddd342bed132f9a"
server_key_iv: "2583d9a00a06b889b7b0951f19ef88b8"
server_key_version: "encryptionkey@0"
I20260619 06:48:37.206921 18732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:48:37.209299 18747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:37.210510 18732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:37.210638 18732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "bdd593b047584f8a801750bcdcdd23f2"
format_stamp: "Formatted at 2026-06-19 06:48:37 on dist-test-slave-9x3b"
server_key: "addf6b8451d492fd4ddd342bed132f9a"
server_key_iv: "2583d9a00a06b889b7b0951f19ef88b8"
server_key_version: "encryptionkey@0"
I20260619 06:48:37.210743 18732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:37.224109 18732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:37.226612 18732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:37.226795 18732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:37.227303 18732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:37.228078 18732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:37.228153 18732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:37.228221 18732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:37.228273 18732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:37.236497 18732 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:33927
I20260619 06:48:37.236521 18860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:33927 every 8 connection(s)
I20260619 06:48:37.237272 18732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:37.243348 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 18732
I20260619 06:48:37.243475 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:37.243815 14803 external_mini_cluster.cc:1428] Setting key 87f541ae7bfeb8d767f71e01c73905b0
Jun 19 06:48:37 dist-test-slave-9x3b krb5kdc[18357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851717, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:37.250193 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:37.238876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:40555 (local address 127.14.116.254:43591)
0619 06:48:37.239001 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:37.239005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:37.239783 (+ 778us) server_negotiation.cc:407] Connection header received
0619 06:48:37.241129 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:37.241132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:37.241188 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:37.241296 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:37.242381 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.242901 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.244502 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.244817 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.244900 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:37.247326 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:37.247329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:37.247331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:37.247354 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:37.248594 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:37.249022 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:37.249024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:37.249025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:37.249094 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:37.249356 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:37.249358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:37.249359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:37.249518 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:37.249587 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:37.249993 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 06:48:37.250083 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:48:37.250991 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:37.239212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:43591 (local address 127.14.116.195:40555)
0619 06:48:37.239636 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:37.239675 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:37.240918 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:37.241463 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:37.241469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:37.241790 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:37.242233 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:37.242245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.243019 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.243023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:37.244366 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:37.244375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.244971 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.244975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:37.245160 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:37.245774 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:37.245810 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:37.247193 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:37.248686 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:37.248690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:37.248700 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:37.248941 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:37.249167 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:37.249169 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:37.249170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:37.249284 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:37.249596 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:37.249600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:37.249799 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 06:48:37.249952 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 06:48:37.250174 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260619 06:48:37.252022 18861 heartbeater.cc:347] Connected to a master server at 127.14.116.254:43591
I20260619 06:48:37.252241 18861 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:37.252681 18861 heartbeater.cc:511] Master 127.14.116.254:43591 requested a full tablet report, sending...
I20260619 06:48:37.253621 18402 ts_manager.cc:194] Registered new tserver with Master: bdd593b047584f8a801750bcdcdd23f2 (127.14.116.195:33927)
I20260619 06:48:37.254164 18402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:40555
I20260619 06:48:37.257303 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:48:37 dist-test-slave-9x3b krb5kdc[18357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:48:37 dist-test-slave-9x3b krb5kdc[18357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:37.281679 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:37.272650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.14.116.193:40235)
0619 06:48:37.273011 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:37.273020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:37.273043 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:48:37.273099 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:37.273104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:37.273320 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:37.273436 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:37.274054 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.274882 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.275739 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.276097 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.276188 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:37.278465 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:37.278473 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:37.278486 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:37.278507 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:37.280041 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:37.280468 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:37.280473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:37.280474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:37.280564 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:37.280789 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:37.280791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:37.280792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:37.281003 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:37.281171 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:37.281338 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 06:48:37.281501 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":162,"threads_started":1}
Jun 19 06:48:37 dist-test-slave-9x3b krb5kdc[18357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:37.291687 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:37.284520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.14.116.254:43591)
0619 06:48:37.284649 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:37.284653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:37.284724 (+ 71us) server_negotiation.cc:407] Connection header received
0619 06:48:37.284816 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:37.284818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:37.284855 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:37.284922 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:37.285448 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.285933 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.286619 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:37.286827 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:37.286896 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:37.289029 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:37.289032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:37.289034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:37.289054 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:37.290345 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:37.290684 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:37.290686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:37.290687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:37.290752 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:37.290964 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:37.290966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:37.290967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:37.291154 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:37.291249 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:37.291419 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 06:48:37.291502 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 06:48:37.292102 18402 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38612
I20260619 06:48:37.292618 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 18464
I20260619 06:48:37.297930 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 18598
I20260619 06:48:37.303006 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 18732
I20260619 06:48:37.308068 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 18370
2026-06-19T06:48:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2066 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:37 dist-test-slave-9x3b krb5kdc[18873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:37 dist-test-slave-9x3b krb5kdc[18873](info): set up 2 sockets
Jun 19 06:48:37 dist-test-slave-9x3b krb5kdc[18873](info): commencing operation
krb5kdc: starting...
W20260619 06:48:38.331758 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:38 dist-test-slave-9x3b krb5kdc[18873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:38.444799 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46257
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:46257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:38.548826 18886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:38.549059 18886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:38.549099 18886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:38.552915 18886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:38.552959 18886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:38.552978 18886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:38.552994 18886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:38.553009 18886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:38.557904 18886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:46257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:38.558766 18886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:38.559738 18886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:38.567595 18892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:38.567595 18894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:38.567659 18891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:38.567924 18886 server_base.cc:1034] running on GCE node
I20260619 06:48:38.568502 18886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:38.569654 18886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:38.570863 18886 hybrid_clock.cc:648] HybridClock initialized: now 1781851718570851 us; error 30 us; skew 500 ppm
Jun 19 06:48:38 dist-test-slave-9x3b krb5kdc[18873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851718, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:38.577126 18886 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:38.577821 18886 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:38.577925 18886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:38.578155 18886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:38.580282 18886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "39299336aa9241b4aebd0a807f36b1bb"
format_stamp: "Formatted at 2026-06-19 06:48:38 on dist-test-slave-9x3b"
server_key: "d408635644b6efb302ee0da8ac84b38b"
server_key_iv: "74458cd11a4547b39131a106648eff49"
server_key_version: "encryptionkey@0"
I20260619 06:48:38.580827 18886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "39299336aa9241b4aebd0a807f36b1bb"
format_stamp: "Formatted at 2026-06-19 06:48:38 on dist-test-slave-9x3b"
server_key: "d408635644b6efb302ee0da8ac84b38b"
server_key_iv: "74458cd11a4547b39131a106648eff49"
server_key_version: "encryptionkey@0"
I20260619 06:48:38.584417 18886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:38.586683 18900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:38.587910 18886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:38.588055 18886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "39299336aa9241b4aebd0a807f36b1bb"
format_stamp: "Formatted at 2026-06-19 06:48:38 on dist-test-slave-9x3b"
server_key: "d408635644b6efb302ee0da8ac84b38b"
server_key_iv: "74458cd11a4547b39131a106648eff49"
server_key_version: "encryptionkey@0"
I20260619 06:48:38.588157 18886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:38.602609 18886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:38.605780 18886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:38.605970 18886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:38.612794 18886 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:46257
I20260619 06:48:38.613045 18952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:46257 every 8 connection(s)
I20260619 06:48:38.613432 18886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:38.615948 18953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:38.619930 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 18886
I20260619 06:48:38.620076 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:38.620349 14803 external_mini_cluster.cc:1428] Setting key fe22497c6e9cc59928c4278286ae99a1
I20260619 06:48:38.621194 18953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb: Bootstrap starting.
I20260619 06:48:38.623353 18953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:38.624106 18953 log.cc:824] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:38.625779 18953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb: No bootstrap required, opened a new log
Jun 19 06:48:38 dist-test-slave-9x3b krb5kdc[18873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:38.629312 18953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39299336aa9241b4aebd0a807f36b1bb" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46257 } }
I20260619 06:48:38.629516 18953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:38.629549 18953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39299336aa9241b4aebd0a807f36b1bb, State: Initialized, Role: FOLLOWER
I20260619 06:48:38.630007 18953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39299336aa9241b4aebd0a807f36b1bb" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46257 } }
I20260619 06:48:38.630119 18953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:38.630196 18953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:38.630290 18953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:38.631078 18953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39299336aa9241b4aebd0a807f36b1bb" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46257 } }
I20260619 06:48:38.631402 18953 leader_election.cc:302] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39299336aa9241b4aebd0a807f36b1bb; no voters:
I20260619 06:48:38.631744 18953 leader_election.cc:288] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:38.631907 18958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:38.632131 18958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [term 1 LEADER]: Becoming Leader. State: Replica: 39299336aa9241b4aebd0a807f36b1bb, State: Running, Role: LEADER
I20260619 06:48:38.632493 18958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39299336aa9241b4aebd0a807f36b1bb" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46257 } }
I20260619 06:48:38.632937 18953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:38.633628 18958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39299336aa9241b4aebd0a807f36b1bb. Latest consensus state: current_term: 1 leader_uuid: "39299336aa9241b4aebd0a807f36b1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39299336aa9241b4aebd0a807f36b1bb" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46257 } } }
I20260619 06:48:38.633810 18958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:38.634246 18967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:38.634730 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:38.621655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.14.116.254:46257)
0619 06:48:38.622031 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:38.622045 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:38.622078 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:48:38.622710 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:38.622747 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:38.623046 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:38.623373 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:38.624166 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:38.625161 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:38.626060 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:38.626528 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:38.626700 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:38.629333 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:38.629339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:38.629361 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:38.629386 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:38.631227 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:38.631724 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:38.631729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:38.631735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:38.631932 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:38.632268 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:38.632272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:38.632273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:38.632752 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:38.632933 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:38.633407 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 06:48:38.633714 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260619 06:48:38.635167 18959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39299336aa9241b4aebd0a807f36b1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39299336aa9241b4aebd0a807f36b1bb" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46257 } } }
I20260619 06:48:38.635294 18959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:38.637666 18967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:38.642799 18967 catalog_manager.cc:1347] Generated new cluster ID: 5788eac795df4e458a607376d01eefb3
I20260619 06:48:38.642858 18967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:38.653203 18967 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:38.654399 18967 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:38.667003 18967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb: Generated new TSK 0
I20260619 06:48:38.667799 18967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:38.692587 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46257
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:38.801347 18979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:38.801582 18979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:38.801625 18979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:38.805456 18979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:38.805498 18979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:38.805554 18979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:38.810638 18979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:38.811509 18979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:38.812417 18979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:38.821033 18987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:38.821038 18985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:38.821043 18984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:38.821306 18979 server_base.cc:1034] running on GCE node
I20260619 06:48:38.821733 18979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:38.822312 18979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:38.823475 18979 hybrid_clock.cc:648] HybridClock initialized: now 1781851718823458 us; error 29 us; skew 500 ppm
Jun 19 06:48:38 dist-test-slave-9x3b krb5kdc[18873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851718, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:38.826160 18979 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:38.826890 18979 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:38.826951 18979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:38.827190 18979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:38.829270 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "9460b4646c434e90b99fda7e01abd57c"
format_stamp: "Formatted at 2026-06-19 06:48:38 on dist-test-slave-9x3b"
server_key: "3e7c7ae97cd758e9a46b74865ef94c29"
server_key_iv: "720120dc1e427f1ad3782bfc2c8b680c"
server_key_version: "encryptionkey@0"
I20260619 06:48:38.829849 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "9460b4646c434e90b99fda7e01abd57c"
format_stamp: "Formatted at 2026-06-19 06:48:38 on dist-test-slave-9x3b"
server_key: "3e7c7ae97cd758e9a46b74865ef94c29"
server_key_iv: "720120dc1e427f1ad3782bfc2c8b680c"
server_key_version: "encryptionkey@0"
I20260619 06:48:38.833258 18979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 06:48:38.835585 18993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:38.836711 18979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:38.836824 18979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "9460b4646c434e90b99fda7e01abd57c"
format_stamp: "Formatted at 2026-06-19 06:48:38 on dist-test-slave-9x3b"
server_key: "3e7c7ae97cd758e9a46b74865ef94c29"
server_key_iv: "720120dc1e427f1ad3782bfc2c8b680c"
server_key_version: "encryptionkey@0"
I20260619 06:48:38.836931 18979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:38.852360 18979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:38.854957 18979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:38.855093 18979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:38.855536 18979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:38.856338 18979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:38.856386 18979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:38.856424 18979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:38.856446 18979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:38.864558 18979 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:34055
I20260619 06:48:38.864615 19106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:34055 every 8 connection(s)
I20260619 06:48:38.865128 18979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:38.868177 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 18979
I20260619 06:48:38.868322 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:38.868610 14803 external_mini_cluster.cc:1428] Setting key 145650c356fd72c38e415eac74d36603
Jun 19 06:48:38 dist-test-slave-9x3b krb5kdc[18873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851718, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
I20260619 06:48:38.879928 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:38.866742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:56997 (local address 127.14.116.254:46257)
0619 06:48:38.866881 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:38.866885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:38.867691 (+ 806us) server_negotiation.cc:407] Connection header received
0619 06:48:38.869107 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:38.869113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:38.869165 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:38.869257 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:38.870593 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:38.871455 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:38.873147 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:38.873404 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:38.873517 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:38.876366 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:38.876369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:38.876371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:38.876393 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:38.877869 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:38.878508 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:38.878511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:38.878512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:38.878584 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:38.878903 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:38.878906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:38.878907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:38.879066 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:38.879136 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:38.879712 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 06:48:38.879809 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
I20260619 06:48:38.880937 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:38.867102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46257 (local address 127.14.116.193:56997)
0619 06:48:38.867524 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:38.867563 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:38.868859 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:38.869469 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:38.869478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:38.869946 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:38.870441 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:38.870455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:38.871643 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:38.871649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:38.873028 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:38.873038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:38.873548 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:38.873552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:38.873712 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:38.874323 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:38.874340 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:38.876177 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:38.877988 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:38.877996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:38.878013 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:38.878347 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:38.878683 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:38.878686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:38.878687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:38.878815 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:38.879150 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:38.879155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:38.879445 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 06:48:38.879591 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 06:48:38.879900 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260619 06:48:38.882117 19107 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46257
I20260619 06:48:38.882395 19107 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:38.882822 19107 heartbeater.cc:511] Master 127.14.116.254:46257 requested a full tablet report, sending...
I20260619 06:48:38.884264 18917 ts_manager.cc:194] Registered new tserver with Master: 9460b4646c434e90b99fda7e01abd57c (127.14.116.193:34055)
I20260619 06:48:38.885351 18917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:56997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:38.894873 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46257
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:39.001579 19112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:39.001827 19112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:39.001871 19112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:39.005739 19112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:39.005785 19112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:39.005841 19112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:39.010967 19112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:39.011842 19112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:39.012717 19112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:39.021586 19117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.021579 19118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.021561 19120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:39.021838 19112 server_base.cc:1034] running on GCE node
I20260619 06:48:39.022225 19112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:39.022794 19112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:39.023967 19112 hybrid_clock.cc:648] HybridClock initialized: now 1781851719023943 us; error 46 us; skew 500 ppm
Jun 19 06:48:39 dist-test-slave-9x3b krb5kdc[18873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851719, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:39.026579 19112 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:39.027275 19112 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:39.027364 19112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:39.027560 19112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:39.029421 19112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "f25b85ef7f8944eb8671ab7a121379b5"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "e26c27476a9fd205471176773da0cbf0"
server_key_iv: "049f26806f6ee19c1a5f809c225f8ec8"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.029973 19112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "f25b85ef7f8944eb8671ab7a121379b5"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "e26c27476a9fd205471176773da0cbf0"
server_key_iv: "049f26806f6ee19c1a5f809c225f8ec8"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.033411 19112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:39.035769 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.036888 19112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:39.037012 19112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "f25b85ef7f8944eb8671ab7a121379b5"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "e26c27476a9fd205471176773da0cbf0"
server_key_iv: "049f26806f6ee19c1a5f809c225f8ec8"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.037112 19112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:39.052651 19112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:39.055163 19112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:39.055348 19112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:39.055912 19112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:39.056663 19112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:39.056732 19112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.056797 19112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:39.056849 19112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.067011 19112 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:40815
I20260619 06:48:39.067039 19239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:40815 every 8 connection(s)
I20260619 06:48:39.067596 19112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:39.070591 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 19112
I20260619 06:48:39.070719 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:39.071002 14803 external_mini_cluster.cc:1428] Setting key c8460d6d40b5f82f6d3b5c5d178ae1da
Jun 19 06:48:39 dist-test-slave-9x3b krb5kdc[18873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851719, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:48:39.081538 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.069386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:52521 (local address 127.14.116.254:46257)
0619 06:48:39.069522 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.069525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.070305 (+ 780us) server_negotiation.cc:407] Connection header received
0619 06:48:39.071744 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.071748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.071801 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.071888 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:39.073061 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.073578 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.074700 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.075084 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.075201 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.077796 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:39.077803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:39.077807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:39.077844 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:39.079491 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:39.080020 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:39.080026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:39.080031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:39.080124 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:39.080450 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:39.080455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:39.080460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:39.080703 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:39.080814 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.081258 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.081366 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:48:39.082291 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.069747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46257 (local address 127.14.116.194:52521)
0619 06:48:39.070168 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:39.070207 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:39.071459 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:39.072041 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:39.072048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:39.072465 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:39.072924 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.072937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.073700 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.073710 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.074580 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.074589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.075223 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.075226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.075374 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.076004 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:39.076028 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:39.077633 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:39.079600 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:39.079604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:39.079660 (+ 56us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:39.079913 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:39.080220 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:39.080222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:39.080223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:39.080350 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:39.080804 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:39.080809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:39.081037 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 06:48:39.081182 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 06:48:39.081435 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:48:39.083503 19240 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46257
I20260619 06:48:39.083779 19240 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:39.084221 19240 heartbeater.cc:511] Master 127.14.116.254:46257 requested a full tablet report, sending...
I20260619 06:48:39.085227 18917 ts_manager.cc:194] Registered new tserver with Master: f25b85ef7f8944eb8671ab7a121379b5 (127.14.116.194:40815)
I20260619 06:48:39.086051 18917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:52521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:39.096457 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46257
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:39.202890 19245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:39.203125 19245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:39.203167 19245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:39.207084 19245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:39.207135 19245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:39.207231 19245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:39.212332 19245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:39.213263 19245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:39.214218 19245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:39.223026 19253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.223110 19250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.223083 19251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:39.223201 19245 server_base.cc:1034] running on GCE node
I20260619 06:48:39.223716 19245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:39.224272 19245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:39.225430 19245 hybrid_clock.cc:648] HybridClock initialized: now 1781851719225402 us; error 39 us; skew 500 ppm
Jun 19 06:48:39 dist-test-slave-9x3b krb5kdc[18873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851719, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:39.228197 19245 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:39.228940 19245 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:39.229031 19245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:39.229235 19245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:39.231194 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "ccc783253b6d42138e871b21abc285cd"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "01abf4b765963714fbdbde7e4da72542"
server_key_iv: "e9e12a65a4418cc3afe13bd45c6a1871"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.231796 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "ccc783253b6d42138e871b21abc285cd"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "01abf4b765963714fbdbde7e4da72542"
server_key_iv: "e9e12a65a4418cc3afe13bd45c6a1871"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.235348 19245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:48:39.237653 19259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.238777 19245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:39.238904 19245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "ccc783253b6d42138e871b21abc285cd"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "01abf4b765963714fbdbde7e4da72542"
server_key_iv: "e9e12a65a4418cc3afe13bd45c6a1871"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.239015 19245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:39.261636 19245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:39.264873 19245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:39.265080 19245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:39.265640 19245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:39.266424 19245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:39.266505 19245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.266572 19245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:39.266621 19245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.274719 19245 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:43317
I20260619 06:48:39.274741 19372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:43317 every 8 connection(s)
I20260619 06:48:39.275259 19245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:39.281986 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 19245
I20260619 06:48:39.282079 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:39.282429 14803 external_mini_cluster.cc:1428] Setting key 2b81de9d4fbc1d3ed1f1f454678d0f68
Jun 19 06:48:39 dist-test-slave-9x3b krb5kdc[18873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851719, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:39.288189 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.276902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:57071 (local address 127.14.116.254:46257)
0619 06:48:39.277035 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.277039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.277758 (+ 719us) server_negotiation.cc:407] Connection header received
0619 06:48:39.278965 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.278969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.279027 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.279132 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:39.280230 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.280746 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.281876 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.282186 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.282277 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.284858 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:39.284861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:39.284863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:39.284885 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:39.286444 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:39.286850 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:39.286852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:39.286854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:39.286925 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:39.287191 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:39.287193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:39.287195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:39.287348 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:39.287425 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.287896 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.288015 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:48:39.288903 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.277197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46257 (local address 127.14.116.195:57071)
0619 06:48:39.277610 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:39.277649 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:39.278756 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:39.279299 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:39.279305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:39.279652 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:39.280089 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.280102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.280865 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.280868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.281713 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.281722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.282336 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.282340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.282538 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.283112 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:39.283138 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:39.284725 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:39.286536 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:39.286540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:39.286549 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:39.286770 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:39.287000 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:39.287002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:39.287004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:39.287118 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:39.287437 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:39.287442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:39.287701 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 06:48:39.287847 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 06:48:39.288096 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260619 06:48:39.289916 19373 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46257
I20260619 06:48:39.290136 19373 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:39.290544 19373 heartbeater.cc:511] Master 127.14.116.254:46257 requested a full tablet report, sending...
I20260619 06:48:39.291440 18917 ts_manager.cc:194] Registered new tserver with Master: ccc783253b6d42138e871b21abc285cd (127.14.116.195:43317)
I20260619 06:48:39.292088 18917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:57071
I20260619 06:48:39.295833 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:48:39.304152 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.297604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.14.116.254:46257)
0619 06:48:39.297722 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.297725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.297909 (+ 184us) server_negotiation.cc:407] Connection header received
0619 06:48:39.297993 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.297995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.298028 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.298084 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:39.298709 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.299186 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.300010 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.300206 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.300281 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.301426 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:39.301429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:39.301430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:39.301449 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:39.302815 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:39.303155 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:39.303157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:39.303158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:39.303228 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:39.303447 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:39.303449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:39.303450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:39.303695 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:39.303793 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.303973 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.304052 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:39.306666 18917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:39.309731 18917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:39.322917 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.317237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:40815 (local address 127.0.0.1:35600)
0619 06:48:39.317560 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:39.317604 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:39.317714 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:39.318937 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:39.318943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:39.318962 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:39.319064 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.319069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.320337 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.320343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.321490 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.321499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.322370 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.322374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.322482 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.322517 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:48:39.322646 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 06:48:39.322758 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
I20260619 06:48:39.322913 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.317336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.14.116.194:40815)
0619 06:48:39.318338 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.318354 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.318478 (+ 124us) server_negotiation.cc:407] Connection header received
0619 06:48:39.318559 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.318565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.318763 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.318928 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:39.319180 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.320224 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.321606 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.322260 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.322383 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.322518 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.322672 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.322757 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":107,"threads_started":1}
I20260619 06:48:39.323371 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.317319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:34055 (local address 127.0.0.1:48550)
0619 06:48:39.318165 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:39.318178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:39.318253 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:39.319253 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:39.319255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:39.319273 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:39.319368 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.319376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.320746 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.320750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.321987 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.321996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.323008 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.323012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.323097 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.323120 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:48:39.323167 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 06:48:39.323229 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":61,"spinlock_wait_cycles":3328,"thread_start_us":105,"threads_started":1}
I20260619 06:48:39.323912 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.317953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:43317 (local address 127.0.0.1:49916)
0619 06:48:39.319190 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:39.319204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:39.319294 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:39.319790 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:39.319793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:39.319812 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:39.319909 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.319914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.321218 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.321221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.322291 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.322300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.323486 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.323492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.323584 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.323647 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 06:48:39.323716 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 06:48:39.323777 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":1,"spinlock_wait_cycles":2944,"thread_start_us":93,"threads_started":1}
I20260619 06:48:39.324100 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.318015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.14.116.193:34055)
0619 06:48:39.318804 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.318819 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.318842 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:48:39.318915 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.318920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.319098 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.319278 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:39.319455 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.320605 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.322104 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.322872 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.323687 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.323812 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.323857 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.323941 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":125,"threads_started":1}
I20260619 06:48:39.324708 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.318384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.14.116.195:43317)
0619 06:48:39.318686 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.318694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.319384 (+ 690us) server_negotiation.cc:407] Connection header received
0619 06:48:39.319458 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.319463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.319659 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.319800 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:39.320027 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.321080 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.322419 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.323209 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.323309 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.323424 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.324485 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.324570 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":131,"threads_started":1}
I20260619 06:48:39.326114 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0adefe89c44bf78e7b663182d13493 (DEFAULT_TABLE table=test-table [id=f9841272fc294a7b9fe53c5312548350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:39.326403 19041 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0adefe89c44bf78e7b663182d13493 (DEFAULT_TABLE table=test-table [id=f9841272fc294a7b9fe53c5312548350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:39.327205 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0adefe89c44bf78e7b663182d13493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:39.327309 19041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0adefe89c44bf78e7b663182d13493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:39.327163 19307 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0adefe89c44bf78e7b663182d13493 (DEFAULT_TABLE table=test-table [id=f9841272fc294a7b9fe53c5312548350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:39.328104 19307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0adefe89c44bf78e7b663182d13493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:39.333025 19390 tablet_bootstrap.cc:492] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c: Bootstrap starting.
I20260619 06:48:39.334417 19391 tablet_bootstrap.cc:492] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: Bootstrap starting.
I20260619 06:48:39.335414 19390 tablet_bootstrap.cc:654] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:39.335879 19392 tablet_bootstrap.cc:492] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: Bootstrap starting.
I20260619 06:48:39.336813 19391 tablet_bootstrap.cc:654] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:39.336817 19390 log.cc:824] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:39.337759 19392 tablet_bootstrap.cc:654] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:39.338007 19391 log.cc:824] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:39.338786 19390 tablet_bootstrap.cc:492] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c: No bootstrap required, opened a new log
I20260619 06:48:39.338905 19392 log.cc:824] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:39.339005 19390 ts_tablet_manager.cc:1397] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:48:39.340051 19391 tablet_bootstrap.cc:492] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: No bootstrap required, opened a new log
I20260619 06:48:39.340296 19391 ts_tablet_manager.cc:1397] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:48:39.340601 19392 tablet_bootstrap.cc:492] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: No bootstrap required, opened a new log
I20260619 06:48:39.340780 19392 ts_tablet_manager.cc:1397] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:48:39.342887 19390 raft_consensus.cc:348] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } }
I20260619 06:48:39.343144 19390 raft_consensus.cc:374] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:39.343226 19390 raft_consensus.cc:729] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9460b4646c434e90b99fda7e01abd57c, State: Initialized, Role: FOLLOWER
I20260619 06:48:39.343741 19390 consensus_queue.cc:260] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } }
I20260619 06:48:39.344187 19391 raft_consensus.cc:348] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } }
I20260619 06:48:39.344375 19107 heartbeater.cc:503] Master 127.14.116.254:46257 was elected leader, sending a full tablet report...
I20260619 06:48:39.344265 19392 raft_consensus.cc:348] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } }
I20260619 06:48:39.344481 19391 raft_consensus.cc:374] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:39.344558 19391 raft_consensus.cc:729] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25b85ef7f8944eb8671ab7a121379b5, State: Initialized, Role: FOLLOWER
I20260619 06:48:39.344575 19392 raft_consensus.cc:374] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:39.344648 19392 raft_consensus.cc:729] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc783253b6d42138e871b21abc285cd, State: Initialized, Role: FOLLOWER
I20260619 06:48:39.344708 19390 ts_tablet_manager.cc:1428] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:48:39.345016 19391 consensus_queue.cc:260] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } }
I20260619 06:48:39.345109 19392 consensus_queue.cc:260] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } }
I20260619 06:48:39.345642 19240 heartbeater.cc:503] Master 127.14.116.254:46257 was elected leader, sending a full tablet report...
I20260619 06:48:39.345870 19391 ts_tablet_manager.cc:1428] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:48:39.345974 19392 ts_tablet_manager.cc:1428] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 06:48:39.346982 19373 heartbeater.cc:503] Master 127.14.116.254:46257 was elected leader, sending a full tablet report...
I20260619 06:48:39.359228 19398 raft_consensus.cc:482] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:39.359400 19398 raft_consensus.cc:504] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } }
I20260619 06:48:39.360401 19398 leader_election.cc:288] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f25b85ef7f8944eb8671ab7a121379b5 (127.14.116.194:40815), 9460b4646c434e90b99fda7e01abd57c (127.14.116.193:34055)
I20260619 06:48:39.363996 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.360585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:40815 (local address 127.14.116.195:54293)
0619 06:48:39.360757 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:39.360772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:39.360869 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:39.361142 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:39.361146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:39.361162 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:39.361288 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.361294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.362061 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.362065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.362874 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.362882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.363522 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.363526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.363671 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.363712 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:48:39.363771 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 06:48:39.363858 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":3}
I20260619 06:48:39.364013 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.360664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:54293 (local address 127.14.116.194:40815)
0619 06:48:39.360775 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.360779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.360838 (+ 59us) server_negotiation.cc:407] Connection header received
0619 06:48:39.360985 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.360989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.361050 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.361124 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:39.361395 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.361961 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.363000 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.363406 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.363531 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.363581 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.363808 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.363893 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:48:39.364039 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.360599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:34055 (local address 127.14.116.195:48607)
0619 06:48:39.360977 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:39.360990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:39.361085 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:39.361311 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:39.361314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:39.361326 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:39.361463 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.361469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.362161 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.362163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.362910 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:39.362918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.363524 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.363528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:39.363722 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.363740 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 06:48:39.363833 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:48:39.363893 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":67,"thread_start_us":96,"threads_started":1}
I20260619 06:48:39.364035 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.360703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:48607 (local address 127.14.116.193:34055)
0619 06:48:39.360816 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.360822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.361068 (+ 246us) server_negotiation.cc:407] Connection header received
0619 06:48:39.361164 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.361167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.361208 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.361292 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:39.361563 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.362058 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.363047 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.363409 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.363528 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.363575 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.363860 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.363936 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:48:39.364692 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0adefe89c44bf78e7b663182d13493" candidate_uuid: "ccc783253b6d42138e871b21abc285cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f25b85ef7f8944eb8671ab7a121379b5" is_pre_election: true
I20260619 06:48:39.364692 19061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0adefe89c44bf78e7b663182d13493" candidate_uuid: "ccc783253b6d42138e871b21abc285cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9460b4646c434e90b99fda7e01abd57c" is_pre_election: true
I20260619 06:48:39.364996 19061 raft_consensus.cc:2454] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc783253b6d42138e871b21abc285cd in term 0.
I20260619 06:48:39.365017 19194 raft_consensus.cc:2454] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc783253b6d42138e871b21abc285cd in term 0.
I20260619 06:48:39.365429 19262 leader_election.cc:302] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9460b4646c434e90b99fda7e01abd57c, ccc783253b6d42138e871b21abc285cd; no voters:
I20260619 06:48:39.365688 19398 raft_consensus.cc:2781] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:39.365767 19398 raft_consensus.cc:482] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:39.365809 19398 raft_consensus.cc:3037] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 0 FOLLOWER]: Advancing to term 1
W20260619 06:48:39.366082 19108 tablet.cc:2369] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:39.366852 19398 raft_consensus.cc:504] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } }
I20260619 06:48:39.367223 19398 leader_election.cc:288] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [CANDIDATE]: Term 1 election: Requested vote from peers f25b85ef7f8944eb8671ab7a121379b5 (127.14.116.194:40815), 9460b4646c434e90b99fda7e01abd57c (127.14.116.193:34055)
I20260619 06:48:39.367524 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0adefe89c44bf78e7b663182d13493" candidate_uuid: "ccc783253b6d42138e871b21abc285cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f25b85ef7f8944eb8671ab7a121379b5"
I20260619 06:48:39.367672 19194 raft_consensus.cc:3037] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:39.368683 19194 raft_consensus.cc:2454] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc783253b6d42138e871b21abc285cd in term 1.
I20260619 06:48:39.368938 19061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0adefe89c44bf78e7b663182d13493" candidate_uuid: "ccc783253b6d42138e871b21abc285cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9460b4646c434e90b99fda7e01abd57c"
I20260619 06:48:39.369019 19261 leader_election.cc:302] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccc783253b6d42138e871b21abc285cd, f25b85ef7f8944eb8671ab7a121379b5; no voters:
I20260619 06:48:39.369063 19061 raft_consensus.cc:3037] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:39.369197 19398 raft_consensus.cc:2781] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:39.369426 19398 raft_consensus.cc:686] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 1 LEADER]: Becoming Leader. State: Replica: ccc783253b6d42138e871b21abc285cd, State: Running, Role: LEADER
I20260619 06:48:39.369748 19398 consensus_queue.cc:237] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } }
I20260619 06:48:39.370292 19061 raft_consensus.cc:2454] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc783253b6d42138e871b21abc285cd in term 1.
I20260619 06:48:39.372844 18916 catalog_manager.cc:5495] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd reported cstate change: term changed from 0 to 1, leader changed from <none> to ccc783253b6d42138e871b21abc285cd (127.14.116.195). New cstate: current_term: 1 leader_uuid: "ccc783253b6d42138e871b21abc285cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccc783253b6d42138e871b21abc285cd" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43317 } health_report { overall_health: HEALTHY } } }
I20260619 06:48:39.387820 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.384373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.14.116.195:43317)
0619 06:48:39.384508 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.384512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.384526 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:39.384689 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.384692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.384740 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.384842 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:39.385117 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.385601 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.386450 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:39.386660 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:39.386766 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:39.387054 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:39.387418 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:39.387544 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.387621 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.387671 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:48:39.393160 19061 raft_consensus.cc:1261] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c [term 1 FOLLOWER]: Refusing update from remote peer ccc783253b6d42138e871b21abc285cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:39.393162 19194 raft_consensus.cc:1261] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [term 1 FOLLOWER]: Refusing update from remote peer ccc783253b6d42138e871b21abc285cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:39.393795 19398 consensus_queue.cc:1035] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9460b4646c434e90b99fda7e01abd57c" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:39.393961 19400 consensus_queue.cc:1035] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f25b85ef7f8944eb8671ab7a121379b5" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:39.400666 19403 mvcc.cc:205] Tried to move back new op lower bound from 7298464642627743744 to 7298464642541584384. Current Snapshot: MvccSnapshot[applied={T|T < 7298464642627743744}]
I20260619 06:48:39.404804 19405 mvcc.cc:205] Tried to move back new op lower bound from 7298464642627743744 to 7298464642541584384. Current Snapshot: MvccSnapshot[applied={T|T < 7298464642627743744}]
I20260619 06:48:39.406195 19404 mvcc.cc:205] Tried to move back new op lower bound from 7298464642627743744 to 7298464642541584384. Current Snapshot: MvccSnapshot[applied={T|T < 7298464642627743744}]
I20260619 06:48:39.410709 18916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:39.410845 18916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:39.413021 18916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39299336aa9241b4aebd0a807f36b1bb: Sending DeleteTablet for 3 replicas of tablet 5f0adefe89c44bf78e7b663182d13493
I20260619 06:48:39.413712 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0adefe89c44bf78e7b663182d13493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:39 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:35600
I20260619 06:48:39.413761 19041 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0adefe89c44bf78e7b663182d13493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:39 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:48550
I20260619 06:48:39.414165 19307 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0adefe89c44bf78e7b663182d13493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:39 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:49916
I20260619 06:48:39.414361 19414 tablet_replica.cc:330] T 5f0adefe89c44bf78e7b663182d13493 P 9460b4646c434e90b99fda7e01abd57c: stopping tablet replica
I20260619 06:48:39.414577 19415 tablet_replica.cc:330] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: stopping tablet replica
I20260619 06:48:39.414590 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 18979
I20260619 06:48:39.415160 19413 tablet_replica.cc:330] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: stopping tablet replica
I20260619 06:48:39.415449 19413 raft_consensus.cc:2229] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:39.415752 19413 raft_consensus.cc:2258] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:39.416141 19415 raft_consensus.cc:2229] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:39.416510 19415 raft_consensus.cc:2258] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:39.417977 19413 ts_tablet_manager.cc:1905] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:39.417989 19415 ts_tablet_manager.cc:1905] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:39.420691 19413 ts_tablet_manager.cc:1918] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:39.420841 19413 log.cc:1196] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/5f0adefe89c44bf78e7b663182d13493
I20260619 06:48:39.421233 19413 ts_tablet_manager.cc:1939] T 5f0adefe89c44bf78e7b663182d13493 P f25b85ef7f8944eb8671ab7a121379b5: Deleting consensus metadata
I20260619 06:48:39.422063 19415 ts_tablet_manager.cc:1918] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:39.422159 19415 log.cc:1196] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/5f0adefe89c44bf78e7b663182d13493
I20260619 06:48:39.422477 19415 ts_tablet_manager.cc:1939] T 5f0adefe89c44bf78e7b663182d13493 P ccc783253b6d42138e871b21abc285cd: Deleting consensus metadata
I20260619 06:48:39.422561 18902 catalog_manager.cc:4841] TS f25b85ef7f8944eb8671ab7a121379b5 (127.14.116.194:40815): tablet 5f0adefe89c44bf78e7b663182d13493 (table test-table [id=f9841272fc294a7b9fe53c5312548350]) successfully deleted
W20260619 06:48:39.423054 18903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:34055 (error 108)
I20260619 06:48:39.423269 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 19112
I20260619 06:48:39.424436 18901 catalog_manager.cc:4841] TS ccc783253b6d42138e871b21abc285cd (127.14.116.195:43317): tablet 5f0adefe89c44bf78e7b663182d13493 (table test-table [id=f9841272fc294a7b9fe53c5312548350]) successfully deleted
I20260619 06:48:39.424474 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.424198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:34055 (local address 127.0.0.1:48562)
0619 06:48:39.424320 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:39.424386 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:34055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 06:48:39.424788 18903 catalog_manager.cc:4584] TS 9460b4646c434e90b99fda7e01abd57c (127.14.116.193:34055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f0adefe89c44bf78e7b663182d13493: Network error: Client connection negotiation failed: client connection to 127.14.116.193:34055: connect: Connection refused (error 111)
I20260619 06:48:39.429518 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 19245
I20260619 06:48:39.435251 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 18886
2026-06-19T06:48:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2127 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T06:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:39Z Disabled control of system clock
I20260619 06:48:39.463912 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:39237
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:39237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:48:39.569994 19420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:39.570266 19420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:39.570303 19420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:39.574240 19420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:39.574293 19420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:39.574309 19420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:39.574326 19420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:39.574340 19420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:39.578977 19420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:39237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:39237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:39.580037 19420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:39.581310 19420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:39.589581 19425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.589564 19426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.589560 19428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:39.589727 19420 server_base.cc:1034] running on GCE node
I20260619 06:48:39.590378 19420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:39.591480 19420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:39.592664 19420 hybrid_clock.cc:648] HybridClock initialized: now 1781851719592649 us; error 26 us; skew 500 ppm
I20260619 06:48:39.594506 19420 webserver.cc:460] Webserver started at http://127.14.116.254:34649/ using document root <none> and password file <none>
I20260619 06:48:39.595068 19420 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:39.595161 19420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:39.595395 19420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:39.597416 19420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "7c5e394684e042a9a59de19c4f220d88"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "6f49a9d50c821ed0c72482c6cce180f8"
server_key_iv: "fd2adc5bcb434ab68f459d24b1cea5f7"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.597958 19420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "7c5e394684e042a9a59de19c4f220d88"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "6f49a9d50c821ed0c72482c6cce180f8"
server_key_iv: "fd2adc5bcb434ab68f459d24b1cea5f7"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.601465 19420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:39.603708 19434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.604815 19420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:39.604948 19420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "7c5e394684e042a9a59de19c4f220d88"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "6f49a9d50c821ed0c72482c6cce180f8"
server_key_iv: "fd2adc5bcb434ab68f459d24b1cea5f7"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.605053 19420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:39.609418 19420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:39.609961 19420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:39.610127 19420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:39.616411 19420 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:39237
I20260619 06:48:39.616415 19486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:39237 every 8 connection(s)
I20260619 06:48:39.617214 19420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:39.618708 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 19420
I20260619 06:48:39.618840 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:39.619107 14803 external_mini_cluster.cc:1428] Setting key 456383ff26a834faed0ea8ece6cbaad2
I20260619 06:48:39.620421 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:39.624231 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.620338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.14.116.254:39237)
0619 06:48:39.620950 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.620964 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.620997 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:48:39.621609 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.621624 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.621946 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.622205 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:39.623005 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:39.623010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:39.623020 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:39.623029 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:39.623077 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:39.623092 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:39.623193 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.623295 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.623534 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":157,"threads_started":1}
I20260619 06:48:39.626328 19487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88: Bootstrap starting.
I20260619 06:48:39.628331 19487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:39.629096 19487 log.cc:824] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:39.630767 19487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88: No bootstrap required, opened a new log
I20260619 06:48:39.633992 19487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e394684e042a9a59de19c4f220d88" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39237 } }
I20260619 06:48:39.634186 19487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:39.634280 19487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5e394684e042a9a59de19c4f220d88, State: Initialized, Role: FOLLOWER
I20260619 06:48:39.634747 19487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e394684e042a9a59de19c4f220d88" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39237 } }
I20260619 06:48:39.634902 19487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:39.634972 19487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:39.635076 19487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:39.635957 19487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e394684e042a9a59de19c4f220d88" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39237 } }
I20260619 06:48:39.636263 19487 leader_election.cc:302] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c5e394684e042a9a59de19c4f220d88; no voters:
I20260619 06:48:39.636538 19487 leader_election.cc:288] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:39.636618 19492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:39.636868 19492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [term 1 LEADER]: Becoming Leader. State: Replica: 7c5e394684e042a9a59de19c4f220d88, State: Running, Role: LEADER
I20260619 06:48:39.637171 19492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e394684e042a9a59de19c4f220d88" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39237 } }
I20260619 06:48:39.637666 19487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:39.638792 19494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c5e394684e042a9a59de19c4f220d88. Latest consensus state: current_term: 1 leader_uuid: "7c5e394684e042a9a59de19c4f220d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e394684e042a9a59de19c4f220d88" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39237 } } }
I20260619 06:48:39.638792 19493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c5e394684e042a9a59de19c4f220d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e394684e042a9a59de19c4f220d88" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39237 } } }
I20260619 06:48:39.638936 19494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:39.638936 19493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:39.639357 19501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:39.642015 19501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:39.646822 19501 catalog_manager.cc:1347] Generated new cluster ID: 72bed371582946c0b2a7c390ba56bd1d
I20260619 06:48:39.646880 19501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:39.670665 19501 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:39.671002 19501 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:39.678364 19501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88: Generated new TSK 0
I20260619 06:48:39.679030 19501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:48:39.693789 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39237
--builtin_ntp_servers=127.14.116.212:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:48:39.814710 19511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:39.814939 19511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:39.814977 19511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:39.818908 19511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:39.818955 19511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:39.819042 19511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:39.823720 19511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:39.824714 19511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:39.826027 19511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:39.835162 19516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.835156 19519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.835156 19517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:39.835309 19511 server_base.cc:1034] running on GCE node
I20260619 06:48:39.835774 19511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:39.836345 19511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:39.837524 19511 hybrid_clock.cc:648] HybridClock initialized: now 1781851719837510 us; error 33 us; skew 500 ppm
I20260619 06:48:39.839241 19511 webserver.cc:460] Webserver started at http://127.14.116.193:43487/ using document root <none> and password file <none>
I20260619 06:48:39.839882 19511 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:39.839970 19511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:39.840165 19511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:39.842074 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "4f442609c631498c9e48c0028cee05e8"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "02501c118c7965342959c1059767d822"
server_key_iv: "2285b7a636f31f65da34758f9951c422"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.842617 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "4f442609c631498c9e48c0028cee05e8"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "02501c118c7965342959c1059767d822"
server_key_iv: "2285b7a636f31f65da34758f9951c422"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.846019 19511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:39.848373 19525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.849642 19511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:39.849749 19511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "4f442609c631498c9e48c0028cee05e8"
format_stamp: "Formatted at 2026-06-19 06:48:39 on dist-test-slave-9x3b"
server_key: "02501c118c7965342959c1059767d822"
server_key_iv: "2285b7a636f31f65da34758f9951c422"
server_key_version: "encryptionkey@0"
I20260619 06:48:39.849854 19511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:39.854283 19511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:39.854825 19511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:39.855001 19511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:39.855499 19511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:39.856335 19511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:39.856411 19511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.856477 19511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:39.856525 19511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:39.864727 19511 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:41905
I20260619 06:48:39.864784 19638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:41905 every 8 connection(s)
I20260619 06:48:39.865548 19511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:39.871071 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.867191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:47387 (local address 127.14.116.254:39237)
0619 06:48:39.867362 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:39.867366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:39.868360 (+ 994us) server_negotiation.cc:407] Connection header received
0619 06:48:39.869541 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:39.869543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:39.869578 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:39.869648 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:39.870655 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:39.870657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:39.870659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:39.870667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:39.870680 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:39.870686 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:39.870748 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:39.870843 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 06:48:39.870885 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:48:39.871169 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 19511
I20260619 06:48:39.871295 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:39.871591 14803 external_mini_cluster.cc:1428] Setting key 287a363ba6534f1e0373eb2fbd4df208
I20260619 06:48:39.871902 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:39.867516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:39237 (local address 127.14.116.193:47387)
0619 06:48:39.868182 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:39.868229 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:39.869372 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:39.869768 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:39.869774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:39.869854 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:39.870436 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:48:39.870437 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:39.870474 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:48:39.870538 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:39.870742 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:39.870746 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:39.870750 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:48:39.870861 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 06:48:39.871028 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":132,"threads_started":1}
I20260619 06:48:39.873080 19639 heartbeater.cc:347] Connected to a master server at 127.14.116.254:39237
I20260619 06:48:39.873296 19639 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:39.873574 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39237
--builtin_ntp_servers=127.14.116.212:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:48:39.873678 19639 heartbeater.cc:511] Master 127.14.116.254:39237 requested a full tablet report, sending...
I20260619 06:48:39.875072 19451 ts_manager.cc:194] Registered new tserver with Master: 4f442609c631498c9e48c0028cee05e8 (127.14.116.193:41905)
W20260619 06:48:39.979736 19642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:39.979970 19642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:39.980007 19642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:39.983943 19642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:39.983989 19642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:39.984041 19642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:39.988536 19642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:39.989476 19642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:39.990684 19642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:39.999601 19647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.999634 19650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:39.999635 19648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:39.999922 19642 server_base.cc:1034] running on GCE node
I20260619 06:48:40.000316 19642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:40.000936 19642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:40.002137 19642 hybrid_clock.cc:648] HybridClock initialized: now 1781851720002119 us; error 29 us; skew 500 ppm
I20260619 06:48:40.004141 19642 webserver.cc:460] Webserver started at http://127.14.116.194:35791/ using document root <none> and password file <none>
I20260619 06:48:40.004755 19642 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:40.004853 19642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:40.005100 19642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:40.007261 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "54190539fb0e4602a9c9afa55a63253f"
format_stamp: "Formatted at 2026-06-19 06:48:40 on dist-test-slave-9x3b"
server_key: "fce3a2b34376c0712c664be75734a457"
server_key_iv: "9a392a0e5aa9403930122a333c770818"
server_key_version: "encryptionkey@0"
I20260619 06:48:40.007897 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "54190539fb0e4602a9c9afa55a63253f"
format_stamp: "Formatted at 2026-06-19 06:48:40 on dist-test-slave-9x3b"
server_key: "fce3a2b34376c0712c664be75734a457"
server_key_iv: "9a392a0e5aa9403930122a333c770818"
server_key_version: "encryptionkey@0"
I20260619 06:48:40.012128 19642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:48:40.014823 19656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:40.016165 19642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:40.016307 19642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "54190539fb0e4602a9c9afa55a63253f"
format_stamp: "Formatted at 2026-06-19 06:48:40 on dist-test-slave-9x3b"
server_key: "fce3a2b34376c0712c664be75734a457"
server_key_iv: "9a392a0e5aa9403930122a333c770818"
server_key_version: "encryptionkey@0"
I20260619 06:48:40.016446 19642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:40.021270 19642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:40.021919 19642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:40.022105 19642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:40.022647 19642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:40.023572 19642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:40.023675 19642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:40.023763 19642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:40.023818 19642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:40.032529 19642 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:36503
I20260619 06:48:40.032550 19769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:36503 every 8 connection(s)
I20260619 06:48:40.033399 19642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:40.039206 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.035090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:36561 (local address 127.14.116.254:39237)
0619 06:48:40.035240 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.035244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.036283 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 06:48:40.037518 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.037520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.037580 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.037686 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.038751 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.038755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.038757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.038769 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.038793 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.038801 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.038900 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.039035 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.039077 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:48:40.039654 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 19642
I20260619 06:48:40.039772 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:40.040063 14803 external_mini_cluster.cc:1428] Setting key d6c98899695cea5b064c61cd7d1e8e7d
I20260619 06:48:40.040285 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.035459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:39237 (local address 127.14.116.194:36561)
0619 06:48:40.036161 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:40.036201 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:40.037338 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:40.037853 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:40.037860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:40.037945 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:40.038462 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:48:40.038463 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:40.038501 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:48:40.038601 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:40.038887 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:40.038895 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:40.038900 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:48:40.039074 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 06:48:40.039296 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":185,"threads_started":1}
I20260619 06:48:40.041831 19770 heartbeater.cc:347] Connected to a master server at 127.14.116.254:39237
I20260619 06:48:40.041977 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39237
--builtin_ntp_servers=127.14.116.212:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:48:40.042055 19770 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:40.042490 19770 heartbeater.cc:511] Master 127.14.116.254:39237 requested a full tablet report, sending...
I20260619 06:48:40.043535 19450 ts_manager.cc:194] Registered new tserver with Master: 54190539fb0e4602a9c9afa55a63253f (127.14.116.194:36503)
W20260619 06:48:40.155498 19773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:40.155889 19773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:40.155972 19773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:40.160073 19773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:40.160147 19773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:40.160235 19773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:40.164922 19773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:40.165984 19773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:40.167528 19773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:40.177748 19779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:40.177731 19778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:40.177932 19773 server_base.cc:1034] running on GCE node
W20260619 06:48:40.177731 19781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:40.178416 19773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:40.179050 19773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:40.180259 19773 hybrid_clock.cc:648] HybridClock initialized: now 1781851720180244 us; error 25 us; skew 500 ppm
I20260619 06:48:40.182121 19773 webserver.cc:460] Webserver started at http://127.14.116.195:39559/ using document root <none> and password file <none>
I20260619 06:48:40.182698 19773 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:40.182793 19773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:40.183022 19773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:40.185056 19773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "bf862ebae3024c8186f2de5e4440d995"
format_stamp: "Formatted at 2026-06-19 06:48:40 on dist-test-slave-9x3b"
server_key: "b02af3f5fbb2dffaea7adfd2e3295a49"
server_key_iv: "e714844f000bb138d517bc97db3f6615"
server_key_version: "encryptionkey@0"
I20260619 06:48:40.185606 19773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "bf862ebae3024c8186f2de5e4440d995"
format_stamp: "Formatted at 2026-06-19 06:48:40 on dist-test-slave-9x3b"
server_key: "b02af3f5fbb2dffaea7adfd2e3295a49"
server_key_iv: "e714844f000bb138d517bc97db3f6615"
server_key_version: "encryptionkey@0"
I20260619 06:48:40.189327 19773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:40.191812 19787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:40.192962 19773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:40.193089 19773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "bf862ebae3024c8186f2de5e4440d995"
format_stamp: "Formatted at 2026-06-19 06:48:40 on dist-test-slave-9x3b"
server_key: "b02af3f5fbb2dffaea7adfd2e3295a49"
server_key_iv: "e714844f000bb138d517bc97db3f6615"
server_key_version: "encryptionkey@0"
I20260619 06:48:40.193199 19773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:40.198019 19773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:40.198602 19773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:40.198774 19773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:40.199271 19773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:40.200191 19773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:40.200256 19773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:40.200320 19773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:40.200371 19773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:40.208918 19773 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:44847
I20260619 06:48:40.208953 19900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:44847 every 8 connection(s)
I20260619 06:48:40.209846 19773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:40.215720 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.211547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:44071 (local address 127.14.116.254:39237)
0619 06:48:40.211704 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.211709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.212718 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 06:48:40.213995 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.213998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.214041 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.214121 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.215203 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.215207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.215210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.215222 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.215245 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.215257 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.215358 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.215511 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.215555 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:48:40.216477 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.211968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:39237 (local address 127.14.116.195:44071)
0619 06:48:40.212577 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:40.212614 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:40.213791 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:40.214285 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:40.214292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:40.214377 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:40.214953 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:48:40.214954 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:40.214992 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:48:40.215056 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:40.215351 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:40.215359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:40.215366 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:48:40.215542 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 06:48:40.215787 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
I20260619 06:48:40.217482 19901 heartbeater.cc:347] Connected to a master server at 127.14.116.254:39237
I20260619 06:48:40.217674 19901 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:40.218017 19901 heartbeater.cc:511] Master 127.14.116.254:39237 requested a full tablet report, sending...
I20260619 06:48:40.218461 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 19773
I20260619 06:48:40.218603 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:40.218914 14803 external_mini_cluster.cc:1428] Setting key 9a00d9dfd198f5d0c050f5f8c9037063
I20260619 06:48:40.219070 19450 ts_manager.cc:194] Registered new tserver with Master: bf862ebae3024c8186f2de5e4440d995 (127.14.116.195:44847)
I20260619 06:48:40.220924 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:48:40.224448 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.222715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.14.116.254:39237)
0619 06:48:40.222829 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.222832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.223042 (+ 210us) server_negotiation.cc:407] Connection header received
0619 06:48:40.223172 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.223175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.223207 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.223297 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.224079 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.224082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.224085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.224096 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.224113 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.224122 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.224214 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.224299 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.224336 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:48:40.226796 19451 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:40.229739 19451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:40.240761 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.237444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:41905 (local address 127.0.0.1:40926)
0619 06:48:40.237873 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:40.237917 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:40.238010 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:40.238938 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:40.238944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:40.239020 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:40.239745 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:48:40.239748 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:40.239770 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:48:40.239838 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:40.240427 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:40.240437 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:40.240442 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:48:40.240553 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 06:48:40.240633 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":69,"threads_started":1}
I20260619 06:48:40.240811 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.237559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.14.116.193:41905)
0619 06:48:40.238452 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.238468 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.238493 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:48:40.238585 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.238592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.238772 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.239308 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.240221 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.240225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.240233 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.240242 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.240271 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.240289 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.240433 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.240606 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.240675 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":130,"threads_started":1}
I20260619 06:48:40.241065 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.237516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:44847 (local address 127.0.0.1:49194)
0619 06:48:40.237922 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:40.237937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:40.238013 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:40.238938 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:40.238944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:40.239020 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:40.239768 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:48:40.239769 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:40.239776 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:48:40.239838 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:40.240480 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:40.240483 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:40.240486 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:48:40.240565 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:48:40.240943 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":312,"thread_start_us":97,"threads_started":1}
I20260619 06:48:40.241108 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.238278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.14.116.194:36503)
0619 06:48:40.238723 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.238734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.238941 (+ 207us) server_negotiation.cc:407] Connection header received
0619 06:48:40.239112 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.239121 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.239323 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.239430 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.240551 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.240556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.240565 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.240577 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.240617 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.240647 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.240739 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.240911 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.240977 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":115,"threads_started":1}
I20260619 06:48:40.241436 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.238128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:36503 (local address 127.0.0.1:43698)
0619 06:48:40.238855 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:40.238867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:40.238996 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:40.239890 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:40.239893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:40.239907 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:40.240402 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:48:40.240409 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:40.240422 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:48:40.240447 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:40.240778 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:40.240781 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:40.240784 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:48:40.241287 (+ 503us) client_negotiation.cc:240] Negotiation successful
0619 06:48:40.241329 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":82,"threads_started":1}
I20260619 06:48:40.241520 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.238221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.14.116.195:44847)
0619 06:48:40.238556 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.238570 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.238587 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:48:40.238627 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.238631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.238772 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.239308 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.240198 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.240203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.240214 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.240224 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.240265 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.240289 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.240434 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.240669 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.240738 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":149,"threads_started":1}
I20260619 06:48:40.243542 19573 tablet_service.cc:1467] Processing CreateTablet for tablet 969823eb11ea432d8a4f31a41bd2d203 (DEFAULT_TABLE table=test-table [id=c42b44add0024cc8af8d759e5884a0a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:40.243583 19835 tablet_service.cc:1467] Processing CreateTablet for tablet 969823eb11ea432d8a4f31a41bd2d203 (DEFAULT_TABLE table=test-table [id=c42b44add0024cc8af8d759e5884a0a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:40.244515 19835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969823eb11ea432d8a4f31a41bd2d203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:40.244514 19573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969823eb11ea432d8a4f31a41bd2d203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:40.250043 19918 tablet_bootstrap.cc:492] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8: Bootstrap starting.
I20260619 06:48:40.249646 19704 tablet_service.cc:1467] Processing CreateTablet for tablet 969823eb11ea432d8a4f31a41bd2d203 (DEFAULT_TABLE table=test-table [id=c42b44add0024cc8af8d759e5884a0a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:40.250660 19704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969823eb11ea432d8a4f31a41bd2d203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:40.251340 19919 tablet_bootstrap.cc:492] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: Bootstrap starting.
I20260619 06:48:40.252449 19918 tablet_bootstrap.cc:654] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:40.253278 19918 log.cc:824] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:40.253404 19919 tablet_bootstrap.cc:654] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:40.254101 19919 log.cc:824] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:40.255374 19918 tablet_bootstrap.cc:492] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8: No bootstrap required, opened a new log
I20260619 06:48:40.255558 19918 ts_tablet_manager.cc:1397] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:48:40.255998 19919 tablet_bootstrap.cc:492] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: No bootstrap required, opened a new log
I20260619 06:48:40.256157 19919 ts_tablet_manager.cc:1397] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:48:40.257589 19922 tablet_bootstrap.cc:492] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: Bootstrap starting.
I20260619 06:48:40.259361 19922 tablet_bootstrap.cc:654] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:40.260074 19922 log.cc:824] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:40.260466 19918 raft_consensus.cc:348] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } }
I20260619 06:48:40.260466 19919 raft_consensus.cc:348] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } }
I20260619 06:48:40.260726 19918 raft_consensus.cc:374] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:40.260726 19919 raft_consensus.cc:374] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:40.260788 19919 raft_consensus.cc:729] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf862ebae3024c8186f2de5e4440d995, State: Initialized, Role: FOLLOWER
I20260619 06:48:40.260788 19918 raft_consensus.cc:729] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f442609c631498c9e48c0028cee05e8, State: Initialized, Role: FOLLOWER
I20260619 06:48:40.261242 19919 consensus_queue.cc:260] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } }
I20260619 06:48:40.261242 19918 consensus_queue.cc:260] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } }
I20260619 06:48:40.261941 19922 tablet_bootstrap.cc:492] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: No bootstrap required, opened a new log
I20260619 06:48:40.262113 19922 ts_tablet_manager.cc:1397] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:48:40.262194 19918 ts_tablet_manager.cc:1428] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260619 06:48:40.262194 19919 ts_tablet_manager.cc:1428] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:48:40.262537 19901 heartbeater.cc:503] Master 127.14.116.254:39237 was elected leader, sending a full tablet report...
I20260619 06:48:40.262720 19639 heartbeater.cc:503] Master 127.14.116.254:39237 was elected leader, sending a full tablet report...
I20260619 06:48:40.266491 19922 raft_consensus.cc:348] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } }
I20260619 06:48:40.266690 19922 raft_consensus.cc:374] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:40.266721 19922 raft_consensus.cc:729] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54190539fb0e4602a9c9afa55a63253f, State: Initialized, Role: FOLLOWER
I20260619 06:48:40.267100 19922 consensus_queue.cc:260] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } }
I20260619 06:48:40.267699 19770 heartbeater.cc:503] Master 127.14.116.254:39237 was elected leader, sending a full tablet report...
I20260619 06:48:40.268000 19922 ts_tablet_manager.cc:1428] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260619 06:48:40.284271 19771 tablet.cc:2369] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:48:40.366662 19640 tablet.cc:2369] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:48:40.460763 19902 tablet.cc:2369] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:40.617288 19925 raft_consensus.cc:482] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:40.617524 19925 raft_consensus.cc:504] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } }
I20260619 06:48:40.618595 19925 leader_election.cc:288] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf862ebae3024c8186f2de5e4440d995 (127.14.116.195:44847), 54190539fb0e4602a9c9afa55a63253f (127.14.116.194:36503)
I20260619 06:48:40.620790 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.618810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:44847 (local address 127.14.116.193:43563)
0619 06:48:40.619223 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:40.619237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:40.619369 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:40.619655 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:40.619658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:40.619672 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:40.620248 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:48:40.620250 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:40.620261 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:48:40.620287 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:40.620527 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:40.620530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:40.620533 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:48:40.620609 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:48:40.620644 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":75,"threads_started":1}
I20260619 06:48:40.620867 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.619009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:43563 (local address 127.14.116.195:44847)
0619 06:48:40.619131 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.619136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.619332 (+ 196us) server_negotiation.cc:407] Connection header received
0619 06:48:40.619482 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.619488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.619533 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.619627 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.620412 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.620415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.620417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.620428 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.620445 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.620453 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.620525 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.620668 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.620715 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 06:48:40.621230 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.618958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:51131 (local address 127.14.116.194:36503)
0619 06:48:40.619082 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.619087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.619388 (+ 301us) server_negotiation.cc:407] Connection header received
0619 06:48:40.619561 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.619567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.619630 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.619703 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.620769 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.620772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.620776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.620787 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.620806 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.620815 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.620874 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.621054 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.621093 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:48:40.621610 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.618812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:36503 (local address 127.14.116.193:51131)
0619 06:48:40.619303 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:40.619316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:40.619446 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:40.619734 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:40.619736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:40.619754 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:40.620605 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:48:40.620609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:40.620619 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:48:40.620648 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:40.620955 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:40.620958 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:40.620961 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:48:40.621441 (+ 480us) client_negotiation.cc:240] Negotiation successful
0619 06:48:40.621479 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":237,"thread_start_us":77,"threads_started":1}
I20260619 06:48:40.621658 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969823eb11ea432d8a4f31a41bd2d203" candidate_uuid: "4f442609c631498c9e48c0028cee05e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf862ebae3024c8186f2de5e4440d995" is_pre_election: true
I20260619 06:48:40.621948 19855 raft_consensus.cc:2454] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f442609c631498c9e48c0028cee05e8 in term 0.
I20260619 06:48:40.622170 19724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969823eb11ea432d8a4f31a41bd2d203" candidate_uuid: "4f442609c631498c9e48c0028cee05e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54190539fb0e4602a9c9afa55a63253f" is_pre_election: true
I20260619 06:48:40.622428 19724 raft_consensus.cc:2454] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f442609c631498c9e48c0028cee05e8 in term 0.
I20260619 06:48:40.622476 19528 leader_election.cc:302] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f442609c631498c9e48c0028cee05e8, bf862ebae3024c8186f2de5e4440d995; no voters:
I20260619 06:48:40.622766 19925 raft_consensus.cc:2781] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:40.622867 19925 raft_consensus.cc:482] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:40.622920 19925 raft_consensus.cc:3037] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:40.623973 19925 raft_consensus.cc:504] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } }
I20260619 06:48:40.624274 19925 leader_election.cc:288] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [CANDIDATE]: Term 1 election: Requested vote from peers bf862ebae3024c8186f2de5e4440d995 (127.14.116.195:44847), 54190539fb0e4602a9c9afa55a63253f (127.14.116.194:36503)
I20260619 06:48:40.624512 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969823eb11ea432d8a4f31a41bd2d203" candidate_uuid: "4f442609c631498c9e48c0028cee05e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf862ebae3024c8186f2de5e4440d995"
I20260619 06:48:40.624610 19724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969823eb11ea432d8a4f31a41bd2d203" candidate_uuid: "4f442609c631498c9e48c0028cee05e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54190539fb0e4602a9c9afa55a63253f"
I20260619 06:48:40.624645 19855 raft_consensus.cc:3037] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:40.624724 19724 raft_consensus.cc:3037] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:40.625890 19855 raft_consensus.cc:2454] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f442609c631498c9e48c0028cee05e8 in term 1.
I20260619 06:48:40.625890 19724 raft_consensus.cc:2454] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f442609c631498c9e48c0028cee05e8 in term 1.
I20260619 06:48:40.626188 19528 leader_election.cc:302] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f442609c631498c9e48c0028cee05e8, bf862ebae3024c8186f2de5e4440d995; no voters:
I20260619 06:48:40.626354 19925 raft_consensus.cc:2781] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:40.626598 19925 raft_consensus.cc:686] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [term 1 LEADER]: Becoming Leader. State: Replica: 4f442609c631498c9e48c0028cee05e8, State: Running, Role: LEADER
I20260619 06:48:40.626897 19925 consensus_queue.cc:237] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } }
I20260619 06:48:40.629745 19451 catalog_manager.cc:5495] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f442609c631498c9e48c0028cee05e8 (127.14.116.193). New cstate: current_term: 1 leader_uuid: "4f442609c631498c9e48c0028cee05e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f442609c631498c9e48c0028cee05e8" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 41905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20260619 06:48:40.692924 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.691655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.14.116.193:41905)
0619 06:48:40.691779 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.691783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.691795 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:48:40.691870 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.691872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.691929 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.691999 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.692574 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.692577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.692579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.692589 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.692606 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.692614 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.692684 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.692722 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.692753 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:48:40.698588 19724 raft_consensus.cc:1261] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [term 1 FOLLOWER]: Refusing update from remote peer 4f442609c631498c9e48c0028cee05e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:40.699218 19855 raft_consensus.cc:1261] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [term 1 FOLLOWER]: Refusing update from remote peer 4f442609c631498c9e48c0028cee05e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:40.699294 19925 consensus_queue.cc:1035] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54190539fb0e4602a9c9afa55a63253f" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:40.699858 19925 consensus_queue.cc:1035] T 969823eb11ea432d8a4f31a41bd2d203 P 4f442609c631498c9e48c0028cee05e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf862ebae3024c8186f2de5e4440d995" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 44847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:40.705920 19932 mvcc.cc:205] Tried to move back new op lower bound from 7298464647974621184 to 7298464647690047488. Current Snapshot: MvccSnapshot[applied={T|T < 7298464647974621184}]
I20260619 06:48:40.707391 19933 mvcc.cc:205] Tried to move back new op lower bound from 7298464647974621184 to 7298464647690047488. Current Snapshot: MvccSnapshot[applied={T|T < 7298464647974621184}]
I20260619 06:48:40.708302 19934 mvcc.cc:205] Tried to move back new op lower bound from 7298464647974621184 to 7298464647690047488. Current Snapshot: MvccSnapshot[applied={T|T < 7298464647974621184}]
I20260619 06:48:40.714566 19451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:40.714727 19451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:40.717069 19451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c5e394684e042a9a59de19c4f220d88: Sending DeleteTablet for 3 replicas of tablet 969823eb11ea432d8a4f31a41bd2d203
I20260619 06:48:40.717780 19835 tablet_service.cc:1514] Processing DeleteTablet for tablet 969823eb11ea432d8a4f31a41bd2d203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:40 UTC) from {username='slave'} at 127.0.0.1:49194
I20260619 06:48:40.717823 19704 tablet_service.cc:1514] Processing DeleteTablet for tablet 969823eb11ea432d8a4f31a41bd2d203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:40 UTC) from {username='slave'} at 127.0.0.1:43698
I20260619 06:48:40.718356 19573 tablet_service.cc:1514] Processing DeleteTablet for tablet 969823eb11ea432d8a4f31a41bd2d203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:40 UTC) from {username='slave'} at 127.0.0.1:40926
I20260619 06:48:40.718588 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 19511
I20260619 06:48:40.718755 19942 tablet_replica.cc:330] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: stopping tablet replica
I20260619 06:48:40.719084 19942 raft_consensus.cc:2229] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:40.719216 19943 tablet_replica.cc:330] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: stopping tablet replica
I20260619 06:48:40.719426 19942 raft_consensus.cc:2258] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:40.719484 19943 raft_consensus.cc:2229] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:40.719833 19943 raft_consensus.cc:2258] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:40.721264 19942 ts_tablet_manager.cc:1905] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:40.721312 19943 ts_tablet_manager.cc:1905] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:40.723541 19943 ts_tablet_manager.cc:1918] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:40.723680 19943 log.cc:1196] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/969823eb11ea432d8a4f31a41bd2d203
I20260619 06:48:40.723964 19943 ts_tablet_manager.cc:1939] T 969823eb11ea432d8a4f31a41bd2d203 P 54190539fb0e4602a9c9afa55a63253f: Deleting consensus metadata
I20260619 06:48:40.724718 19942 ts_tablet_manager.cc:1918] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:40.724838 19942 log.cc:1196] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/969823eb11ea432d8a4f31a41bd2d203
I20260619 06:48:40.724974 19435 catalog_manager.cc:4841] TS 54190539fb0e4602a9c9afa55a63253f (127.14.116.194:36503): tablet 969823eb11ea432d8a4f31a41bd2d203 (table test-table [id=c42b44add0024cc8af8d759e5884a0a9]) successfully deleted
I20260619 06:48:40.725157 19942 ts_tablet_manager.cc:1939] T 969823eb11ea432d8a4f31a41bd2d203 P bf862ebae3024c8186f2de5e4440d995: Deleting consensus metadata
I20260619 06:48:40.726292 19437 catalog_manager.cc:4841] TS bf862ebae3024c8186f2de5e4440d995 (127.14.116.195:44847): tablet 969823eb11ea432d8a4f31a41bd2d203 (table test-table [id=c42b44add0024cc8af8d759e5884a0a9]) successfully deleted
W20260619 06:48:40.726728 19435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:41905 (error 108)
I20260619 06:48:40.726769 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 19642
I20260619 06:48:40.728372 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.728123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:41905 (local address 127.0.0.1:40938)
0619 06:48:40.728245 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:40.728310 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:41905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:48:40.728534 19435 catalog_manager.cc:4584] TS 4f442609c631498c9e48c0028cee05e8 (127.14.116.193:41905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 969823eb11ea432d8a4f31a41bd2d203: Network error: Client connection negotiation failed: client connection to 127.14.116.193:41905: connect: Connection refused (error 111)
I20260619 06:48:40.732264 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 19773
I20260619 06:48:40.737571 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 19420
2026-06-19T06:48:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1301 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 06:48:40.758311 14803 mini_oidc.cc:152] Starting JWKS server
I20260619 06:48:40.759891 14803 webserver.cc:460] Webserver started at https://127.0.0.1:39525/ using document root <none> and password file <none>
I20260619 06:48:40.760066 14803 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:48:40.760356 14803 webserver.cc:460] Webserver started at http://127.0.0.1:35527/ using document root <none> and password file <none>
2026-06-19T06:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:40Z Disabled control of system clock
I20260619 06:48:40.768996 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38467
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:38467
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39525/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/testchainca.cert with env {}
W20260619 06:48:40.875564 19951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:40.875864 19951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:40.875903 19951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:40.879952 19951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:40.880008 19951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:40.880024 19951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:40.880040 19951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:40.880056 19951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:48:40.880071 19951 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:48:40.880085 19951 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39525/jwks.json
I20260619 06:48:40.885308 19951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:38467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39525/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:40.886318 19951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:40.887286 19951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:40.895411 19956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:40.895433 19959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:40.895414 19957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:40.895704 19951 server_base.cc:1034] running on GCE node
I20260619 06:48:40.896409 19951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:40.897496 19951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:40.898679 19951 hybrid_clock.cc:648] HybridClock initialized: now 1781851720898665 us; error 26 us; skew 500 ppm
I20260619 06:48:40.900519 19951 webserver.cc:460] Webserver started at http://127.14.116.254:43027/ using document root <none> and password file <none>
I20260619 06:48:40.901084 19951 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:40.901175 19951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:40.901397 19951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:40.903388 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "89ccc309f10548cdbb74470f490e7ab2"
format_stamp: "Formatted at 2026-06-19 06:48:40 on dist-test-slave-9x3b"
server_key: "5181ad7b97a3d6e2657cf2e3f358e890"
server_key_iv: "9d2bd97990b254811bad2453e1c0c5a5"
server_key_version: "encryptionkey@0"
I20260619 06:48:40.903995 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "89ccc309f10548cdbb74470f490e7ab2"
format_stamp: "Formatted at 2026-06-19 06:48:40 on dist-test-slave-9x3b"
server_key: "5181ad7b97a3d6e2657cf2e3f358e890"
server_key_iv: "9d2bd97990b254811bad2453e1c0c5a5"
server_key_version: "encryptionkey@0"
I20260619 06:48:40.907337 19951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:48:40.909614 19965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:40.910724 19951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:48:40.910873 19951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "89ccc309f10548cdbb74470f490e7ab2"
format_stamp: "Formatted at 2026-06-19 06:48:40 on dist-test-slave-9x3b"
server_key: "5181ad7b97a3d6e2657cf2e3f358e890"
server_key_iv: "9d2bd97990b254811bad2453e1c0c5a5"
server_key_version: "encryptionkey@0"
I20260619 06:48:40.911013 19951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:40.925329 19951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:40.926010 19951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:40.926208 19951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:40.932705 19951 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:38467
I20260619 06:48:40.932718 20017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:38467 every 8 connection(s)
I20260619 06:48:40.933640 19951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:40.933975 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 19951
I20260619 06:48:40.934090 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:40.934470 14803 external_mini_cluster.cc:1428] Setting key 7bab8751bd89fcc84f56d8c9d972c2ba
I20260619 06:48:40.937325 20018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:40.942883 20018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2: Bootstrap starting.
I20260619 06:48:40.943478 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:40.935847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.14.116.254:38467)
0619 06:48:40.936471 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:40.936486 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:40.936522 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:48:40.937180 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:40.937197 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:40.937651 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:40.937988 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:40.938412 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:40.939331 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:40.940287 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:40.940755 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:40.940935 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:40.941790 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:40.941795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:40.941807 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:40.942175 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:40.942231 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:40.942249 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:40.942359 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:40.942507 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 06:48:40.942753 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
I20260619 06:48:40.945053 20018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:40.945896 20018 log.cc:824] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:40.946239 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:48:40.948022 20018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2: No bootstrap required, opened a new log
I20260619 06:48:40.951274 20018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ccc309f10548cdbb74470f490e7ab2" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38467 } }
I20260619 06:48:40.951534 20018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:40.951599 20018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ccc309f10548cdbb74470f490e7ab2, State: Initialized, Role: FOLLOWER
I20260619 06:48:40.952013 20018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ccc309f10548cdbb74470f490e7ab2" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38467 } }
I20260619 06:48:40.952132 20018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:40.952172 20018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:40.952255 20018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:40.953083 20018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ccc309f10548cdbb74470f490e7ab2" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38467 } }
I20260619 06:48:40.953364 20018 leader_election.cc:302] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ccc309f10548cdbb74470f490e7ab2; no voters:
I20260619 06:48:40.953619 20018 leader_election.cc:288] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:40.953742 20023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:40.954013 20023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [term 1 LEADER]: Becoming Leader. State: Replica: 89ccc309f10548cdbb74470f490e7ab2, State: Running, Role: LEADER
I20260619 06:48:40.954348 20023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ccc309f10548cdbb74470f490e7ab2" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38467 } }
I20260619 06:48:40.954849 20018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:40.955943 20025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ccc309f10548cdbb74470f490e7ab2. Latest consensus state: current_term: 1 leader_uuid: "89ccc309f10548cdbb74470f490e7ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ccc309f10548cdbb74470f490e7ab2" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38467 } } }
I20260619 06:48:40.956153 20025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:40.956179 20024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89ccc309f10548cdbb74470f490e7ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ccc309f10548cdbb74470f490e7ab2" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38467 } } }
I20260619 06:48:40.956350 20024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:40.956682 20032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:40.960000 20032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:40.965610 20032 catalog_manager.cc:1347] Generated new cluster ID: fb1af15834d243b0b9991361fdd2a102
I20260619 06:48:40.965761 20032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:40.979152 20032 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:40.980419 20032 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:40.989264 20032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89ccc309f10548cdbb74470f490e7ab2: Generated new TSK 0
I20260619 06:48:40.990094 20032 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:48:41.021718 20048 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 06:48:41.021847 20048 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:48:41.022445 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:41.005152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.14.116.254:38467)
0619 06:48:41.005284 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:41.005289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:41.005477 (+ 188us) server_negotiation.cc:407] Connection header received
0619 06:48:41.005657 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:41.005661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:41.005715 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:41.005823 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:48:41.006055 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.006881 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.007729 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.008087 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.008190 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.008501 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:48:41.022020 (+ 13519us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:48:41.022294 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44798: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 06:48:41.022527 20021 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44798: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 06:48:41.023561 20047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:41.005017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38467 (local address 127.0.0.1:44798)
0619 06:48:41.005395 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:41.005407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:41.005513 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:41.005850 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:41.005852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:41.005864 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:48:41.005960 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:41.005965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.006984 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.006986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:41.007580 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:41.007587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.008202 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.008205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:41.008271 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.008314 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:48:41.022245 (+ 13931us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:48:41.022455 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.116.254:38467: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.116.254:38467: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39525/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 06:48:41.024557 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 19951
2026-06-19T06:48:41Z chronyd exiting
I20260619 06:48:41.060814 14803 test_util.cc:175] -----------------------------------------------
I20260619 06:48:41.061045 14803 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781851695409368-14803-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 06:48:41.062114 14803 mini_oidc.cc:152] Starting JWKS server
I20260619 06:48:41.062973 14803 webserver.cc:460] Webserver started at https://127.0.0.1:45969/ using document root <none> and password file <none>
I20260619 06:48:41.063082 14803 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:48:41.063334 14803 webserver.cc:460] Webserver started at http://127.0.0.1:46869/ using document root <none> and password file <none>
2026-06-19T06:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:41Z Disabled control of system clock
I20260619 06:48:41.073423 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38489
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:38489
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45969/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/testchainca.cert with env {}
W20260619 06:48:41.181468 20054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:41.181717 20054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:41.181783 20054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:41.185845 20054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:41.185899 20054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:41.185915 20054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:41.185932 20054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:41.185969 20054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:48:41.186002 20054 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:48:41.186026 20054 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45969/jwks.json
I20260619 06:48:41.191452 20054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:38489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45969/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:41.192582 20054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:41.193569 20054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:41.201749 20059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:41.201756 20062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:41.201786 20060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:41.202104 20054 server_base.cc:1034] running on GCE node
I20260619 06:48:41.202620 20054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:41.203715 20054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:41.204900 20054 hybrid_clock.cc:648] HybridClock initialized: now 1781851721204893 us; error 31 us; skew 500 ppm
I20260619 06:48:41.206693 20054 webserver.cc:460] Webserver started at http://127.14.116.254:36507/ using document root <none> and password file <none>
I20260619 06:48:41.207245 20054 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:41.207334 20054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:41.207540 20054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:41.209543 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "837a8a665624424e948bb8182a2f121a"
format_stamp: "Formatted at 2026-06-19 06:48:41 on dist-test-slave-9x3b"
server_key: "e05ff90814b1c0c93076b86bc435acda"
server_key_iv: "6d095785182b79f1a66b9020b33cf7de"
server_key_version: "encryptionkey@0"
I20260619 06:48:41.210093 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "837a8a665624424e948bb8182a2f121a"
format_stamp: "Formatted at 2026-06-19 06:48:41 on dist-test-slave-9x3b"
server_key: "e05ff90814b1c0c93076b86bc435acda"
server_key_iv: "6d095785182b79f1a66b9020b33cf7de"
server_key_version: "encryptionkey@0"
I20260619 06:48:41.213639 20054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:41.215951 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:41.217079 20054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:41.217211 20054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "837a8a665624424e948bb8182a2f121a"
format_stamp: "Formatted at 2026-06-19 06:48:41 on dist-test-slave-9x3b"
server_key: "e05ff90814b1c0c93076b86bc435acda"
server_key_iv: "6d095785182b79f1a66b9020b33cf7de"
server_key_version: "encryptionkey@0"
I20260619 06:48:41.217320 20054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:41.231283 20054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:41.231979 20054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:41.232172 20054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:41.238785 20054 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:38489
I20260619 06:48:41.238811 20120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:38489 every 8 connection(s)
I20260619 06:48:41.239699 20054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:41.242689 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:41.248030 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a: Bootstrap starting.
I20260619 06:48:41.248909 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 20054
I20260619 06:48:41.249042 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:41.249316 14803 external_mini_cluster.cc:1428] Setting key ca75d3223e9beae31a5c9241ee1f86f0
I20260619 06:48:41.250761 20121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:41.251483 20121 log.cc:824] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:41.253284 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a: No bootstrap required, opened a new log
I20260619 06:48:41.256749 20121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837a8a665624424e948bb8182a2f121a" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38489 } }
I20260619 06:48:41.256985 20121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:41.257057 20121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837a8a665624424e948bb8182a2f121a, State: Initialized, Role: FOLLOWER
I20260619 06:48:41.257534 20121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837a8a665624424e948bb8182a2f121a" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38489 } }
I20260619 06:48:41.257684 20121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:41.257767 20121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:41.257882 20121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:41.258105 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:41.250319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.14.116.254:38489)
0619 06:48:41.250967 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:41.250981 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:41.251015 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:48:41.251674 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:41.251701 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:41.252079 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:41.252416 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:41.252812 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.253799 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.254697 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.255162 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.255334 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.256267 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:41.256271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:41.256283 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:41.256660 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:41.256712 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:41.256731 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:41.256838 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:41.256987 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 06:48:41.257259 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260619 06:48:41.258965 20121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837a8a665624424e948bb8182a2f121a" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38489 } }
I20260619 06:48:41.259289 20121 leader_election.cc:302] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 837a8a665624424e948bb8182a2f121a; no voters:
I20260619 06:48:41.259604 20121 leader_election.cc:288] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:41.259707 20126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:41.259886 20126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [term 1 LEADER]: Becoming Leader. State: Replica: 837a8a665624424e948bb8182a2f121a, State: Running, Role: LEADER
I20260619 06:48:41.260454 20126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837a8a665624424e948bb8182a2f121a" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38489 } }
I20260619 06:48:41.260845 20121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:41.261114 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:48:41.262560 20128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 837a8a665624424e948bb8182a2f121a. Latest consensus state: current_term: 1 leader_uuid: "837a8a665624424e948bb8182a2f121a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837a8a665624424e948bb8182a2f121a" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38489 } } }
I20260619 06:48:41.262677 20128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:41.262867 20127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "837a8a665624424e948bb8182a2f121a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837a8a665624424e948bb8182a2f121a" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38489 } } }
I20260619 06:48:41.263371 20127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a [sys.catalog]: This master's current role is: LEADER
W20260619 06:48:41.265442 20141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:48:41.265502 20141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:48:41.265578 20140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:41.266248 20140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:41.271304 20140 catalog_manager.cc:1347] Generated new cluster ID: e7a6f3fdd8054e0190aca0298a21698c
I20260619 06:48:41.271366 20140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:41.281749 20140 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:41.282924 20140 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:41.300098 20140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 837a8a665624424e948bb8182a2f121a: Generated new TSK 0
I20260619 06:48:41.301074 20140 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:48:41.335171 20151 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 06:48:41.335264 20151 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:48:41.335920 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:41.320311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.14.116.254:38489)
0619 06:48:41.320442 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:41.320447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:41.320621 (+ 174us) server_negotiation.cc:407] Connection header received
0619 06:48:41.320760 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:41.320764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:41.320818 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:41.320939 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:48:41.321132 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.321954 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.322768 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.323122 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.323222 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.323475 (+ 253us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:48:41.335504 (+ 12029us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:48:41.335772 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 06:48:41.335939 20150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:41.320182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38489 (local address 127.0.0.1:41590)
0619 06:48:41.320538 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:41.320550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:41.320640 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:41.320942 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:41.320944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:41.320950 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:48:41.321037 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:41.321041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.322055 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.322057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:41.322662 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:41.322669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.323230 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.323233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:41.323299 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.323318 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:48:41.335702 (+ 12384us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:48:41.335816 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.116.254:38489: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
W20260619 06:48:41.335985 20124 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 06:48:41.336930 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 20054
2026-06-19T06:48:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (313 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 06:48:41.375689 14803 mini_oidc.cc:152] Starting JWKS server
I20260619 06:48:41.376529 14803 webserver.cc:460] Webserver started at https://127.0.0.1:37223/ using document root <none> and password file <none>
I20260619 06:48:41.376610 14803 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:48:41.376811 14803 webserver.cc:460] Webserver started at http://127.0.0.1:37103/ using document root <none> and password file <none>
2026-06-19T06:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:41Z Disabled control of system clock
I20260619 06:48:41.385250 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38121
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:38121
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37223/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:48:41.489218 20157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:41.489521 20157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:41.489557 20157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:41.493422 20157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:41.493470 20157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:41.493485 20157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:41.493501 20157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:41.493516 20157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:48:41.493531 20157 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:48:41.493544 20157 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37223/jwks.json
I20260619 06:48:41.498140 20157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:38121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37223/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:41.499109 20157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:41.500471 20157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:41.508406 20162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:41.508450 20163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:41.508538 20157 server_base.cc:1034] running on GCE node
W20260619 06:48:41.508450 20165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:41.509133 20157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:41.510259 20157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:41.511420 20157 hybrid_clock.cc:648] HybridClock initialized: now 1781851721511391 us; error 40 us; skew 500 ppm
I20260619 06:48:41.513329 20157 webserver.cc:460] Webserver started at http://127.14.116.254:40049/ using document root <none> and password file <none>
I20260619 06:48:41.513872 20157 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:41.513933 20157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:41.514175 20157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:41.516191 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "f859874ecfa64f8d9597c650d4079b23"
format_stamp: "Formatted at 2026-06-19 06:48:41 on dist-test-slave-9x3b"
server_key: "d06d0397e2e3141439eb1178d20e8b36"
server_key_iv: "52bd7ec7ecd7aa60e28a68350b52846e"
server_key_version: "encryptionkey@0"
I20260619 06:48:41.516760 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "f859874ecfa64f8d9597c650d4079b23"
format_stamp: "Formatted at 2026-06-19 06:48:41 on dist-test-slave-9x3b"
server_key: "d06d0397e2e3141439eb1178d20e8b36"
server_key_iv: "52bd7ec7ecd7aa60e28a68350b52846e"
server_key_version: "encryptionkey@0"
I20260619 06:48:41.520325 20157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:41.522552 20171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:41.523674 20157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:41.523804 20157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "f859874ecfa64f8d9597c650d4079b23"
format_stamp: "Formatted at 2026-06-19 06:48:41 on dist-test-slave-9x3b"
server_key: "d06d0397e2e3141439eb1178d20e8b36"
server_key_iv: "52bd7ec7ecd7aa60e28a68350b52846e"
server_key_version: "encryptionkey@0"
I20260619 06:48:41.523909 20157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:41.540784 20157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:41.541401 20157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:41.541585 20157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:41.548177 20157 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:38121
I20260619 06:48:41.548200 20223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:38121 every 8 connection(s)
I20260619 06:48:41.549065 20157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:41.550246 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 20157
I20260619 06:48:41.550383 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:41.550660 14803 external_mini_cluster.cc:1428] Setting key fa4729bdc8c93e3e13c13b52f824a11c
I20260619 06:48:41.552428 20224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:41.558871 20224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23: Bootstrap starting.
I20260619 06:48:41.560631 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:41.551957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.14.116.254:38121)
0619 06:48:41.552584 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:41.552601 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:41.552642 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:48:41.553431 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:41.553453 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:41.553929 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:41.554284 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:41.554690 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.556181 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.556991 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.557668 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.557845 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.558613 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:41.558619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:41.558634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:41.559142 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:41.559206 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:41.559231 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:41.559316 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:41.559461 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 06:48:41.559729 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260619 06:48:41.561370 20224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:41.562073 20224 log.cc:824] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:41.563211 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:48:41.564234 20224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23: No bootstrap required, opened a new log
I20260619 06:48:41.567484 20224 raft_consensus.cc:348] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f859874ecfa64f8d9597c650d4079b23" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38121 } }
I20260619 06:48:41.567741 20224 raft_consensus.cc:374] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:41.567844 20224 raft_consensus.cc:729] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f859874ecfa64f8d9597c650d4079b23, State: Initialized, Role: FOLLOWER
I20260619 06:48:41.568290 20224 consensus_queue.cc:260] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f859874ecfa64f8d9597c650d4079b23" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38121 } }
I20260619 06:48:41.568434 20224 raft_consensus.cc:388] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:41.568499 20224 raft_consensus.cc:482] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:41.568589 20224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:41.569423 20224 raft_consensus.cc:504] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f859874ecfa64f8d9597c650d4079b23" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38121 } }
I20260619 06:48:41.569749 20224 leader_election.cc:302] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f859874ecfa64f8d9597c650d4079b23; no voters:
I20260619 06:48:41.570031 20224 leader_election.cc:288] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:41.570188 20229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:41.570504 20229 raft_consensus.cc:686] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [term 1 LEADER]: Becoming Leader. State: Replica: f859874ecfa64f8d9597c650d4079b23, State: Running, Role: LEADER
I20260619 06:48:41.570801 20229 consensus_queue.cc:237] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f859874ecfa64f8d9597c650d4079b23" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38121 } }
I20260619 06:48:41.571236 20224 sys_catalog.cc:564] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:41.572412 20231 sys_catalog.cc:455] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f859874ecfa64f8d9597c650d4079b23. Latest consensus state: current_term: 1 leader_uuid: "f859874ecfa64f8d9597c650d4079b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f859874ecfa64f8d9597c650d4079b23" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38121 } } }
I20260619 06:48:41.572557 20231 sys_catalog.cc:458] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:41.572933 20238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:41.572963 20230 sys_catalog.cc:455] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f859874ecfa64f8d9597c650d4079b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f859874ecfa64f8d9597c650d4079b23" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38121 } } }
I20260619 06:48:41.573570 20230 sys_catalog.cc:458] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:41.575578 20238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:41.581264 20238 catalog_manager.cc:1347] Generated new cluster ID: 3af01907918f4a34aaaf38e11a052bb6
I20260619 06:48:41.581331 20238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:41.586144 20238 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:41.587265 20238 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:41.591830 20238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f859874ecfa64f8d9597c650d4079b23: Generated new TSK 0
I20260619 06:48:41.592526 20238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:48:41.608214 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:41.603360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.14.116.254:38121)
0619 06:48:41.603487 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:41.603492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:41.603666 (+ 174us) server_negotiation.cc:407] Connection header received
0619 06:48:41.603840 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:41.603843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:41.603896 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:41.604007 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:48:41.604187 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.604981 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.605783 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.606129 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.606229 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.606480 (+ 251us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:48:41.607858 (+ 1378us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:48:41.608075 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47366: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 06:48:41.608283 20227 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47366: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 06:48:41.610041 20253 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:41.603253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38121 (local address 127.0.0.1:47366)
0619 06:48:41.603558 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:41.603569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:41.603719 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:41.604009 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:41.604011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:41.604017 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:48:41.604098 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:41.604103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.605093 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.605095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:41.605680 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:41.605686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.606238 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.606240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:41.606299 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.606319 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:48:41.608010 (+ 1691us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:48:41.609873 (+ 1863us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.116.254:38121: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1682,"thread_start_us":112,"threads_started":1}
I20260619 06:48:41.610332 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 20157
E20260619 06:48:41.619192 20254 webserver.cc:533] Webserver: sslize failed:
E20260619 06:48:41.619256 20254 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T06:48:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (273 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T06:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:41Z Disabled control of system clock
I20260619 06:48:41.657069 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:33853
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:33853
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/testchainca.cert with env {}
W20260619 06:48:41.762310 20258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:41.762562 20258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:41.762600 20258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:41.766757 20258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:41.766819 20258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:41.766834 20258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:41.766851 20258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:41.766866 20258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:48:41.766884 20258 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:48:41.766898 20258 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 06:48:41.772083 20258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:33853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:41.772986 20258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:41.773887 20258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:41.781989 20263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:41.781965 20264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:41.782074 20258 server_base.cc:1034] running on GCE node
W20260619 06:48:41.781965 20266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:41.782738 20258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:41.783849 20258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:41.785037 20258 hybrid_clock.cc:648] HybridClock initialized: now 1781851721785025 us; error 26 us; skew 500 ppm
I20260619 06:48:41.786827 20258 webserver.cc:460] Webserver started at http://127.14.116.254:34587/ using document root <none> and password file <none>
I20260619 06:48:41.787377 20258 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:41.787469 20258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:41.787747 20258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:41.789675 20258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "1b34420a595544e49c91664fb7d2a623"
format_stamp: "Formatted at 2026-06-19 06:48:41 on dist-test-slave-9x3b"
server_key: "f8e0f609538e707167bee0c76828fce7"
server_key_iv: "9b1345ba72921a68c346cc56396b5376"
server_key_version: "encryptionkey@0"
I20260619 06:48:41.790226 20258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "1b34420a595544e49c91664fb7d2a623"
format_stamp: "Formatted at 2026-06-19 06:48:41 on dist-test-slave-9x3b"
server_key: "f8e0f609538e707167bee0c76828fce7"
server_key_iv: "9b1345ba72921a68c346cc56396b5376"
server_key_version: "encryptionkey@0"
I20260619 06:48:41.793661 20258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:41.795926 20272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:41.797056 20258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:48:41.797194 20258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "1b34420a595544e49c91664fb7d2a623"
format_stamp: "Formatted at 2026-06-19 06:48:41 on dist-test-slave-9x3b"
server_key: "f8e0f609538e707167bee0c76828fce7"
server_key_iv: "9b1345ba72921a68c346cc56396b5376"
server_key_version: "encryptionkey@0"
I20260619 06:48:41.797302 20258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:41.809069 20258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:41.809648 20258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:41.809834 20258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:41.816394 20324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:33853 every 8 connection(s)
I20260619 06:48:41.816395 20258 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:33853
I20260619 06:48:41.817298 20258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:41.819886 20325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:41.821463 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 20258
I20260619 06:48:41.821599 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:41.821892 14803 external_mini_cluster.cc:1428] Setting key d2cadc2379a45a5b4d94caed4202d6cd
I20260619 06:48:41.826172 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623: Bootstrap starting.
I20260619 06:48:41.828780 20325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:41.829555 20325 log.cc:824] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:41.830648 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:41.823042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.14.116.254:33853)
0619 06:48:41.823573 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:41.823586 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:41.823638 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:48:41.824219 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:41.824235 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:41.824603 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:41.824920 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:41.825319 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.826208 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.827420 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.827921 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.828033 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.828832 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:41.828857 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:41.828875 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:48:41.829247 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:41.829298 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:48:41.829317 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:41.829396 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:41.829563 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 06:48:41.829830 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
I20260619 06:48:41.831375 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623: No bootstrap required, opened a new log
I20260619 06:48:41.833292 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:48:41.835525 20325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34420a595544e49c91664fb7d2a623" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33853 } }
I20260619 06:48:41.835781 20325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:41.835825 20325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b34420a595544e49c91664fb7d2a623, State: Initialized, Role: FOLLOWER
I20260619 06:48:41.836165 20325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34420a595544e49c91664fb7d2a623" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33853 } }
I20260619 06:48:41.836268 20325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:41.836309 20325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:41.836374 20325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:41.837139 20325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34420a595544e49c91664fb7d2a623" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33853 } }
I20260619 06:48:41.837482 20325 leader_election.cc:302] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b34420a595544e49c91664fb7d2a623; no voters:
I20260619 06:48:41.837852 20325 leader_election.cc:288] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:41.837941 20330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:41.838201 20330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [term 1 LEADER]: Becoming Leader. State: Replica: 1b34420a595544e49c91664fb7d2a623, State: Running, Role: LEADER
I20260619 06:48:41.838538 20330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34420a595544e49c91664fb7d2a623" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33853 } }
I20260619 06:48:41.838999 20325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:41.840235 20332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b34420a595544e49c91664fb7d2a623. Latest consensus state: current_term: 1 leader_uuid: "1b34420a595544e49c91664fb7d2a623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34420a595544e49c91664fb7d2a623" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33853 } } }
I20260619 06:48:41.840253 20331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b34420a595544e49c91664fb7d2a623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b34420a595544e49c91664fb7d2a623" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33853 } } }
I20260619 06:48:41.840382 20332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:41.840389 20331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:41.840718 20339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:41.843451 20339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:41.848471 20339 catalog_manager.cc:1347] Generated new cluster ID: 747c8fae1ebc4fc1a5e5c571c0a9c704
I20260619 06:48:41.848543 20339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:41.854769 20339 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:41.855540 20339 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:41.859299 20339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b34420a595544e49c91664fb7d2a623: Generated new TSK 0
I20260619 06:48:41.859977 20339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:48:42.294592 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:41.874984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.14.116.254:33853)
0619 06:48:41.875119 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:41.875126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:41.875277 (+ 151us) server_negotiation.cc:407] Connection header received
0619 06:48:41.875441 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:41.875443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:41.875489 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:41.875568 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:48:41.875853 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.876364 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.877616 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.877831 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.877916 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.878218 (+ 302us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:48:42.294199 (+415981us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:48:42.294475 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41396: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 06:48:42.294713 20328 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41396: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 06:48:42.294770 20354 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:41.874904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:33853 (local address 127.0.0.1:41396)
0619 06:48:41.875186 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:41.875201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:41.875311 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:41.875600 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:41.875619 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:41.875629 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:48:41.875747 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:41.875754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.876492 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.876496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:41.877502 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:41.877512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:41.877954 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:41.877959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:41.878048 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:41.878075 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:48:42.294430 (+416355us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:48:42.294592 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.116.254:33853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
I20260619 06:48:42.295547 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 20258
2026-06-19T06:48:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (665 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (116 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:42 dist-test-slave-9x3b krb5kdc[20366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:42 dist-test-slave-9x3b krb5kdc[20366](info): set up 2 sockets
Jun 19 06:48:42 dist-test-slave-9x3b krb5kdc[20366](info): commencing operation
krb5kdc: starting...
W20260619 06:48:43.559897 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:43 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:43.672340 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:40867
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:43.777613 20379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:43.777904 20379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:43.777973 20379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:43.781864 20379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:43.781948 20379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:43.781987 20379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:43.782047 20379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:43.782085 20379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:43.787120 20379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:43.788112 20379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:43.789077 20379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:43.797122 20384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:43.797120 20385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:43.797116 20387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:43.797392 20379 server_base.cc:1034] running on GCE node
I20260619 06:48:43.797921 20379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:43.798944 20379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:43.800096 20379 hybrid_clock.cc:648] HybridClock initialized: now 1781851723800083 us; error 25 us; skew 500 ppm
Jun 19 06:48:43 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851723, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:43.807484 20379 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:43.808431 20379 webserver.cc:460] Webserver started at http://127.14.116.254:42309/ using document root <none> and password file <none>
I20260619 06:48:43.808974 20379 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:43.809062 20379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:43.809289 20379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:43.811229 20379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "0e3b2d8e005f4d81989288016223310d"
format_stamp: "Formatted at 2026-06-19 06:48:43 on dist-test-slave-9x3b"
server_key: "86f190f7802f5477224d5ca0372ced5b"
server_key_iv: "d6ed5c026bee13e32d05509f89dbf3c6"
server_key_version: "encryptionkey@0"
I20260619 06:48:43.811842 20379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "0e3b2d8e005f4d81989288016223310d"
format_stamp: "Formatted at 2026-06-19 06:48:43 on dist-test-slave-9x3b"
server_key: "86f190f7802f5477224d5ca0372ced5b"
server_key_iv: "d6ed5c026bee13e32d05509f89dbf3c6"
server_key_version: "encryptionkey@0"
I20260619 06:48:43.815246 20379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:48:43.817543 20394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:43.818828 20379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:43.818966 20379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "0e3b2d8e005f4d81989288016223310d"
format_stamp: "Formatted at 2026-06-19 06:48:43 on dist-test-slave-9x3b"
server_key: "86f190f7802f5477224d5ca0372ced5b"
server_key_iv: "d6ed5c026bee13e32d05509f89dbf3c6"
server_key_version: "encryptionkey@0"
I20260619 06:48:43.819070 20379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:43.833195 20379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:43.836012 20379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:43.836202 20379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:43.842813 20379 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:40867
I20260619 06:48:43.842839 20446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:40867 every 8 connection(s)
I20260619 06:48:43.843724 20379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:43.846266 20447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:43.847991 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 20379
I20260619 06:48:43.848119 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:43.848412 14803 external_mini_cluster.cc:1428] Setting key acdbbaddaa057e5d0867768a1d06c771
I20260619 06:48:43.852365 20447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d: Bootstrap starting.
I20260619 06:48:43.854447 20447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:43.855173 20447 log.cc:824] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:48:43 dist-test-slave-9x3b krb5kdc[20366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:43.857321 20447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d: No bootstrap required, opened a new log
I20260619 06:48:43.861356 20447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3b2d8e005f4d81989288016223310d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40867 } }
I20260619 06:48:43.861558 20447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:43.861619 20447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3b2d8e005f4d81989288016223310d, State: Initialized, Role: FOLLOWER
I20260619 06:48:43.862043 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:43.849790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.14.116.254:40867)
0619 06:48:43.850197 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:43.850210 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:43.850243 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:48:43.850849 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:43.850864 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:43.851172 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:43.851529 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:43.852317 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:43.853319 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:43.854169 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:43.854686 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:43.854850 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:43.857630 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:43.857635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:43.857646 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:43.857670 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:43.859234 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:43.859745 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:43.859750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:43.859754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:43.859852 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:43.860134 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:43.860136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:43.860137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:43.860466 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:43.860607 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:43.860915 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 06:48:43.861202 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260619 06:48:43.862031 20447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3b2d8e005f4d81989288016223310d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40867 } }
I20260619 06:48:43.862210 20447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:43.862314 20447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:43.862428 20447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:43.863329 20447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3b2d8e005f4d81989288016223310d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40867 } }
I20260619 06:48:43.863683 20447 leader_election.cc:302] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e3b2d8e005f4d81989288016223310d; no voters:
I20260619 06:48:43.864010 20447 leader_election.cc:288] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:43.864069 20452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:43.864289 20452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [term 1 LEADER]: Becoming Leader. State: Replica: 0e3b2d8e005f4d81989288016223310d, State: Running, Role: LEADER
I20260619 06:48:43.864635 20452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3b2d8e005f4d81989288016223310d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40867 } }
I20260619 06:48:43.865214 20447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:43.866202 20453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e3b2d8e005f4d81989288016223310d. Latest consensus state: current_term: 1 leader_uuid: "0e3b2d8e005f4d81989288016223310d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3b2d8e005f4d81989288016223310d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40867 } } }
I20260619 06:48:43.866343 20453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:43.866202 20454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e3b2d8e005f4d81989288016223310d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3b2d8e005f4d81989288016223310d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40867 } } }
I20260619 06:48:43.866583 20454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:43.866714 20461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:43.869629 20461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:43.875121 20461 catalog_manager.cc:1347] Generated new cluster ID: bf3b057d0a7c4885a04e28a4a1774eb3
I20260619 06:48:43.875193 20461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:43.894032 20461 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:43.895221 20461 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:43.900877 20461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e3b2d8e005f4d81989288016223310d: Generated new TSK 0
I20260619 06:48:43.901521 20461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:43.926527 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40867
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:44.032416 20473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:44.032650 20473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:44.032691 20473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:44.036607 20473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:44.036655 20473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:44.036710 20473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:44.041778 20473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:44.042647 20473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:44.043541 20473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:44.052392 20478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:44.052415 20479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:44.052438 20481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:44.052639 20473 server_base.cc:1034] running on GCE node
I20260619 06:48:44.053027 20473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:44.053582 20473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:44.054760 20473 hybrid_clock.cc:648] HybridClock initialized: now 1781851724054752 us; error 28 us; skew 500 ppm
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.057358 20473 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:44.058172 20473 webserver.cc:460] Webserver started at http://127.14.116.193:46625/ using document root <none> and password file <none>
I20260619 06:48:44.058730 20473 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:44.058820 20473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:44.059037 20473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:44.061079 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "5a4bae89ac39450ca4dd09272fa061bb"
format_stamp: "Formatted at 2026-06-19 06:48:44 on dist-test-slave-9x3b"
server_key: "f1628639da1aec057895044548d7e3b2"
server_key_iv: "4241ba75bd264a5ca1a2ffe72a4551fb"
server_key_version: "encryptionkey@0"
I20260619 06:48:44.061632 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "5a4bae89ac39450ca4dd09272fa061bb"
format_stamp: "Formatted at 2026-06-19 06:48:44 on dist-test-slave-9x3b"
server_key: "f1628639da1aec057895044548d7e3b2"
server_key_iv: "4241ba75bd264a5ca1a2ffe72a4551fb"
server_key_version: "encryptionkey@0"
I20260619 06:48:44.065125 20473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:44.067551 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:44.068697 20473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:44.068832 20473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "5a4bae89ac39450ca4dd09272fa061bb"
format_stamp: "Formatted at 2026-06-19 06:48:44 on dist-test-slave-9x3b"
server_key: "f1628639da1aec057895044548d7e3b2"
server_key_iv: "4241ba75bd264a5ca1a2ffe72a4551fb"
server_key_version: "encryptionkey@0"
I20260619 06:48:44.068938 20473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:44.079815 20473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:44.082248 20473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:44.082439 20473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:44.082955 20473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:44.083798 20473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:44.083864 20473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:44.083936 20473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:44.083989 20473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:44.092573 20473 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:44145
I20260619 06:48:44.092600 20601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:44145 every 8 connection(s)
I20260619 06:48:44.093375 20473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:44.102021 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 20473
I20260619 06:48:44.102145 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:44.102412 14803 external_mini_cluster.cc:1428] Setting key db48ac13f030c62f52bf2e6f62fdc998
I20260619 06:48:44.106745 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:44.095024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:53987 (local address 127.14.116.254:40867)
0619 06:48:44.095179 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:44.095184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:44.096039 (+ 855us) server_negotiation.cc:407] Connection header received
0619 06:48:44.097312 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:44.097316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:44.097360 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:44.097440 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:44.098555 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.099074 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.100233 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.100461 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.100538 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:44.102977 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:44.102983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:44.102990 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:44.103011 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:44.104533 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.105162 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.105164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.105166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.105238 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.105666 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.105670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.105671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.105852 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:44.105964 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:44.106454 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 06:48:44.106576 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:48:44.107522 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:44.095407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40867 (local address 127.14.116.193:53987)
0619 06:48:44.095879 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:44.095940 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:44.097121 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:44.097600 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:44.097607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:44.097946 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:44.098432 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:44.098446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.099177 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.099180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:44.100131 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:44.100140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.100589 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.100591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:44.100724 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:44.101315 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:44.101330 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:44.102814 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:44.104658 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:44.104667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:44.104684 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:44.105060 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:44.105324 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:44.105330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:44.105335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:44.105535 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:44.105963 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:44.105970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:44.106216 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 06:48:44.106405 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 06:48:44.106714 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":96,"threads_started":1}
I20260619 06:48:44.108752 20602 heartbeater.cc:347] Connected to a master server at 127.14.116.254:40867
I20260619 06:48:44.109027 20602 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:44.109632 20602 heartbeater.cc:511] Master 127.14.116.254:40867 requested a full tablet report, sending...
I20260619 06:48:44.111325 20411 ts_manager.cc:194] Registered new tserver with Master: 5a4bae89ac39450ca4dd09272fa061bb (127.14.116.193:44145)
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
I20260619 06:48:44.112883 20411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:53987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:44.130146 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40867
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:44.236984 20607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:44.237195 20607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:44.237233 20607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:44.241047 20607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:44.241087 20607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:44.241138 20607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:44.245990 20607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:44.246821 20607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:44.247699 20607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:44.256687 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:44.256762 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:44.256688 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:44.256975 20607 server_base.cc:1034] running on GCE node
I20260619 06:48:44.257467 20607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:44.258024 20607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:44.259168 20607 hybrid_clock.cc:648] HybridClock initialized: now 1781851724259154 us; error 26 us; skew 500 ppm
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.261842 20607 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:44.262653 20607 webserver.cc:460] Webserver started at http://127.14.116.194:36935/ using document root <none> and password file <none>
I20260619 06:48:44.263125 20607 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:44.263180 20607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:44.263340 20607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:44.265308 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "5452f530ab5e40398c8162b17cf95db5"
format_stamp: "Formatted at 2026-06-19 06:48:44 on dist-test-slave-9x3b"
server_key: "76850720d6919ed262a1a40f5be2fcbe"
server_key_iv: "b14dd703722bb57222bf75cfabea5edf"
server_key_version: "encryptionkey@0"
I20260619 06:48:44.265856 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "5452f530ab5e40398c8162b17cf95db5"
format_stamp: "Formatted at 2026-06-19 06:48:44 on dist-test-slave-9x3b"
server_key: "76850720d6919ed262a1a40f5be2fcbe"
server_key_iv: "b14dd703722bb57222bf75cfabea5edf"
server_key_version: "encryptionkey@0"
I20260619 06:48:44.269459 20607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:48:44.271600 20622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:44.272825 20607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:44.272945 20607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "5452f530ab5e40398c8162b17cf95db5"
format_stamp: "Formatted at 2026-06-19 06:48:44 on dist-test-slave-9x3b"
server_key: "76850720d6919ed262a1a40f5be2fcbe"
server_key_iv: "b14dd703722bb57222bf75cfabea5edf"
server_key_version: "encryptionkey@0"
I20260619 06:48:44.273033 20607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:44.290132 20607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:44.293246 20607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:44.293444 20607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:44.293987 20607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:44.294761 20607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:44.294828 20607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:44.294895 20607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:44.294955 20607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:44.303993 20607 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:44897
I20260619 06:48:44.304014 20735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:44897 every 8 connection(s)
I20260619 06:48:44.304811 20607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:44.305712 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 20607
I20260619 06:48:44.305867 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:44.306141 14803 external_mini_cluster.cc:1428] Setting key 5caf2d0afcbbb4f8488b8e2571c8d694
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:48:44.319036 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:44.306563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:34837 (local address 127.14.116.254:40867)
0619 06:48:44.306810 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:44.306815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:44.307675 (+ 860us) server_negotiation.cc:407] Connection header received
0619 06:48:44.309613 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:44.309616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:44.309667 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:44.309768 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:44.310901 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.311490 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.312621 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.312847 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.312954 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:44.315369 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:44.315372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:44.315374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:44.315397 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:44.316729 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.317331 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.317336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.317338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.317442 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.317812 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.317815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.317816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.317990 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:44.318072 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:44.318664 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 06:48:44.318820 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260619 06:48:44.319988 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:44.307009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40867 (local address 127.14.116.194:34837)
0619 06:48:44.307513 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:44.307555 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:44.309391 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:44.309956 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:44.309963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:44.310303 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:44.310753 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:44.310766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.311633 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.311637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:44.312501 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:44.312510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.312989 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.312993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:44.313157 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:44.313771 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:44.313796 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:44.315180 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:44.316860 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:44.316866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:44.316879 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:44.317220 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:44.317534 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:44.317537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:44.317538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:44.317708 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:44.318094 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:44.318103 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:44.318418 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 06:48:44.318622 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 06:48:44.319087 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20260619 06:48:44.321044 20736 heartbeater.cc:347] Connected to a master server at 127.14.116.254:40867
I20260619 06:48:44.321285 20736 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:44.321749 20736 heartbeater.cc:511] Master 127.14.116.254:40867 requested a full tablet report, sending...
I20260619 06:48:44.322712 20411 ts_manager.cc:194] Registered new tserver with Master: 5452f530ab5e40398c8162b17cf95db5 (127.14.116.194:44897)
I20260619 06:48:44.323258 20411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:34837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:44.331975 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40867
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:44.442296 20741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:44.442548 20741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:44.442588 20741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:44.446661 20741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:44.446713 20741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:44.446768 20741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:44.452014 20741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:44.452970 20741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:44.454038 20741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:44.463186 20747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:44.463251 20749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:44.463281 20741 server_base.cc:1034] running on GCE node
W20260619 06:48:44.463186 20746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:44.463930 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:44.464493 20741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:44.465646 20741 hybrid_clock.cc:648] HybridClock initialized: now 1781851724465634 us; error 39 us; skew 500 ppm
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.468384 20741 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:44.469138 20741 webserver.cc:460] Webserver started at http://127.14.116.195:34925/ using document root <none> and password file <none>
I20260619 06:48:44.469657 20741 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:44.469717 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:44.469882 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:44.471827 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "1cd16568a2c8450184041adc7b2041f6"
format_stamp: "Formatted at 2026-06-19 06:48:44 on dist-test-slave-9x3b"
server_key: "339eb880e73f02f76aa8532e2a8f2681"
server_key_iv: "f26d8cd2fac4e6fed1b55f563647ace0"
server_key_version: "encryptionkey@0"
I20260619 06:48:44.472328 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "1cd16568a2c8450184041adc7b2041f6"
format_stamp: "Formatted at 2026-06-19 06:48:44 on dist-test-slave-9x3b"
server_key: "339eb880e73f02f76aa8532e2a8f2681"
server_key_iv: "f26d8cd2fac4e6fed1b55f563647ace0"
server_key_version: "encryptionkey@0"
I20260619 06:48:44.475693 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:44.478056 20756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:44.479184 20741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:44.479346 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "1cd16568a2c8450184041adc7b2041f6"
format_stamp: "Formatted at 2026-06-19 06:48:44 on dist-test-slave-9x3b"
server_key: "339eb880e73f02f76aa8532e2a8f2681"
server_key_iv: "f26d8cd2fac4e6fed1b55f563647ace0"
server_key_version: "encryptionkey@0"
I20260619 06:48:44.479440 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:44.506613 20741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:44.509407 20741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:44.509596 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:44.510174 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:44.510958 20741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:44.511003 20741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:44.511066 20741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:44.511101 20741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:44.519752 20741 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:38667
I20260619 06:48:44.519793 20869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:38667 every 8 connection(s)
I20260619 06:48:44.520603 20741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:44.527794 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 20741
I20260619 06:48:44.527907 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:44.528203 14803 external_mini_cluster.cc:1428] Setting key 19b492aacd1528dd4082790400a50cab
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:44.533660 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:44.522340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:54695 (local address 127.14.116.254:40867)
0619 06:48:44.522468 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:44.522472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:44.523341 (+ 869us) server_negotiation.cc:407] Connection header received
0619 06:48:44.524676 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:44.524679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:44.524736 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:44.524840 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:44.525977 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.526501 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.527630 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.527854 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.527988 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:44.530768 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:44.530770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:44.530772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:44.530794 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:44.532035 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.532480 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.532483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.532484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.532556 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.532825 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.532827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.532828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.532987 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:44.533059 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:44.533461 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 06:48:44.533550 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:44.534453 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:44.522700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40867 (local address 127.14.116.195:54695)
0619 06:48:44.523201 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:44.523242 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:44.524472 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:44.525038 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:44.525044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:44.525397 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:44.525839 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:44.525851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.526634 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.526637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:44.527487 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:44.527495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.528034 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.528038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:44.528254 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:44.528874 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:44.528894 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:44.530628 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:44.532132 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:44.532137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:44.532147 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:44.532387 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:44.532629 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:44.532631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:44.532633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:44.532753 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:44.533065 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:44.533069 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:44.533276 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 06:48:44.533428 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 06:48:44.533656 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":112,"threads_started":1}
I20260619 06:48:44.535439 20870 heartbeater.cc:347] Connected to a master server at 127.14.116.254:40867
I20260619 06:48:44.535686 20870 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:44.536120 20870 heartbeater.cc:511] Master 127.14.116.254:40867 requested a full tablet report, sending...
I20260619 06:48:44.537082 20411 ts_manager.cc:194] Registered new tserver with Master: 1cd16568a2c8450184041adc7b2041f6 (127.14.116.195:38667)
I20260619 06:48:44.537604 20411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:54695
I20260619 06:48:44.541741 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/test-admin.keytab.
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.556545 14803 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 06:48:44.559273 20877 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:44.557865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.14.116.193:44145)
0619 06:48:44.558183 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:44.558192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:44.558224 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:48:44.558272 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:44.558276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:44.558457 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:44.558577 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:44.559103 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41368: BlockingRecv error: recv got EOF from 127.0.0.1:41368 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":130,"threads_started":1}
W20260619 06:48:44.559443 14803 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.560456 14803 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.561332 14803 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:44.570421 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:44.562122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.14.116.193:44145)
0619 06:48:44.562245 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:44.562251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:44.562369 (+ 118us) server_negotiation.cc:407] Connection header received
0619 06:48:44.562446 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:44.562448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:44.562486 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:44.562551 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:44.563022 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.563899 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.564676 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.565081 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.565159 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:44.567456 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:44.567464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:44.567476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:44.567497 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:44.568909 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.569300 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.569305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.569307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.569383 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.569602 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.569604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.569605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.569829 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:44.569979 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:44.570156 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 06:48:44.570273 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 06:48:44.574295 20877 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:44.573213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.14.116.193:44145)
0619 06:48:44.573353 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:44.573357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:44.573461 (+ 104us) server_negotiation.cc:407] Connection header received
0619 06:48:44.573594 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:44.573597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:44.573634 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:44.573717 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:44.574164 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41390: BlockingRecv error: recv got EOF from 127.0.0.1:41390 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 06:48:44.574321 14803 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.575225 14803 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.576175 14803 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:44.584156 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:44.576922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.14.116.193:44145)
0619 06:48:44.577044 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:44.577048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:44.577122 (+ 74us) server_negotiation.cc:407] Connection header received
0619 06:48:44.577197 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:44.577199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:44.577230 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:44.577300 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:44.577722 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.578204 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.578911 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.579115 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.579188 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:44.581230 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:44.581233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:44.581235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:44.581254 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:44.582663 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.583034 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.583037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.583038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.583107 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.583361 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.583363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.583364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.583500 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:44.583589 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:44.583876 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 06:48:44.584013 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 06:48:44.586491 20877 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:44.585664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.14.116.193:44145)
0619 06:48:44.585793 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:44.585796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:44.585830 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:48:44.585930 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:44.585932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:44.585964 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:44.586020 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:44.586400 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41406: BlockingRecv error: recv got EOF from 127.0.0.1:41406 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 06:48:44.586540 14803 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781851695409368-14803-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.587443 14803 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:44.588310 14803 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:48:44.596088 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:44.589094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.14.116.193:44145)
0619 06:48:44.589191 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:44.589195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:44.589302 (+ 107us) server_negotiation.cc:407] Connection header received
0619 06:48:44.589383 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:44.589385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:44.589415 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:44.589477 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:44.589887 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.590367 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.591079 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:44.591273 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:44.591352 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:44.593324 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:44.593327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:44.593329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:44.593349 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:44.594793 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.595184 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.595187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.595188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.595253 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:44.595475 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:44.595477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:44.595478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:44.595664 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:44.595758 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:44.595907 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 06:48:44.595987 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:48:44.596978 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 20473
I20260619 06:48:44.602541 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 20607
I20260619 06:48:44.607398 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 20741
I20260619 06:48:44.612313 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 20379
2026-06-19T06:48:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2081 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20890](info): set up 2 sockets
Jun 19 06:48:44 dist-test-slave-9x3b krb5kdc[20890](info): commencing operation
krb5kdc: starting...
W20260619 06:48:45.640101 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:45 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.254.keytab.
Entry for principal oryx/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.254.keytab.
I20260619 06:48:45.752532 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:40519
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.254.keytab
--principal=oryx/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:45.858525 20903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:45.858826 20903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:45.858896 20903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:45.862903 20903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:45.862990 20903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:45.863029 20903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:45.863085 20903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:45.863121 20903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:45.868407 20903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.254.keytab
--principal=oryx/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:40519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:45.869418 20903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:45.870407 20903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:45.878566 20908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:45.878567 20911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:45.878655 20903 server_base.cc:1034] running on GCE node
W20260619 06:48:45.878567 20909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:45.879374 20903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:45.880491 20903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:45.881709 20903 hybrid_clock.cc:648] HybridClock initialized: now 1781851725881688 us; error 30 us; skew 500 ppm
Jun 19 06:48:45 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851725, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:45.884598 20903 init.cc:377] Logged in from keytab as oryx/127.14.116.254@KRBTEST.COM (short username oryx)
I20260619 06:48:45.885509 20903 webserver.cc:460] Webserver started at http://127.14.116.254:41539/ using document root <none> and password file <none>
I20260619 06:48:45.886049 20903 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:45.886139 20903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:45.886365 20903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:45.888445 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "a21688a106c343d58a6797563ec5ff64"
format_stamp: "Formatted at 2026-06-19 06:48:45 on dist-test-slave-9x3b"
server_key: "4198aea10c176e207e760a29408a8fa5"
server_key_iv: "1d46dafc7dabe41c8ccedcf2eb069358"
server_key_version: "encryptionkey@0"
I20260619 06:48:45.889007 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "a21688a106c343d58a6797563ec5ff64"
format_stamp: "Formatted at 2026-06-19 06:48:45 on dist-test-slave-9x3b"
server_key: "4198aea10c176e207e760a29408a8fa5"
server_key_iv: "1d46dafc7dabe41c8ccedcf2eb069358"
server_key_version: "encryptionkey@0"
I20260619 06:48:45.892905 20903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:48:45.895294 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:45.896483 20903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:45.896623 20903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "a21688a106c343d58a6797563ec5ff64"
format_stamp: "Formatted at 2026-06-19 06:48:45 on dist-test-slave-9x3b"
server_key: "4198aea10c176e207e760a29408a8fa5"
server_key_iv: "1d46dafc7dabe41c8ccedcf2eb069358"
server_key_version: "encryptionkey@0"
I20260619 06:48:45.896726 20903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:45.931473 20903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:45.934803 20903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:45.935001 20903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:45.941949 20903 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:40519
I20260619 06:48:45.942233 20970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:40519 every 8 connection(s)
I20260619 06:48:45.942901 20903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:45.945447 20971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:45.948661 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 20903
I20260619 06:48:45.948801 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:45.949121 14803 external_mini_cluster.cc:1428] Setting key 6bb2848b263d440a545c20036aa0a58f
I20260619 06:48:45.951228 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: Bootstrap starting.
I20260619 06:48:45.953689 20971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:45.954428 20971 log.cc:824] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:45.956437 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: No bootstrap required, opened a new log
Jun 19 06:48:45 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:45.960816 20971 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } }
I20260619 06:48:45.961068 20971 raft_consensus.cc:374] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:45.961126 20971 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21688a106c343d58a6797563ec5ff64, State: Initialized, Role: FOLLOWER
I20260619 06:48:45.961562 20971 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } }
I20260619 06:48:45.961722 20971 raft_consensus.cc:388] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:45.961789 20971 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:45.961882 20971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:45.962951 20971 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } }
I20260619 06:48:45.963285 20971 leader_election.cc:302] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a21688a106c343d58a6797563ec5ff64; no voters:
I20260619 06:48:45.963675 20971 leader_election.cc:288] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:45.963749 20976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:45.963955 20976 raft_consensus.cc:686] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 1 LEADER]: Becoming Leader. State: Replica: a21688a106c343d58a6797563ec5ff64, State: Running, Role: LEADER
I20260619 06:48:45.963990 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:45.950472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.14.116.254:40519)
0619 06:48:45.950886 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:45.950901 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:45.950935 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:48:45.951595 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:45.951657 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:45.951985 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:45.952250 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:45.953118 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:45.954286 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:45.955312 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:45.955820 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:45.955938 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:45.958643 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:45.958649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:45.958664 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:45.958694 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:45.960743 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:45.961236 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:45.961241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:45.961245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:45.961390 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:45.961755 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:45.961759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:45.961760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:45.962229 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:45.962394 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:45.962672 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 06:48:45.962979 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260619 06:48:45.964363 20976 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } }
I20260619 06:48:45.964920 20971 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:45.966135 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } }
I20260619 06:48:45.966127 20978 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21688a106c343d58a6797563ec5ff64. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } }
I20260619 06:48:45.966356 20978 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:45.966356 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
W20260619 06:48:45.969468 20991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:48:45.969564 20991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:48:45.969643 20987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:45.970288 20987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:45.974953 20987 catalog_manager.cc:1347] Generated new cluster ID: 22e17090ec044d4086624ef9fca979b3
I20260619 06:48:45.975016 20987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:45.993141 20987 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:45.994050 20987 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:46.016933 20987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: Generated new TSK 0
I20260619 06:48:46.017664 20987 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab.
Entry for principal oryx/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab.
I20260619 06:48:46.046092 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40519
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab
--principal=oryx/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:46.153343 20997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:46.153615 20997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:46.153687 20997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:46.157693 20997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:46.157775 20997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:46.157864 20997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:46.162868 20997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab
--principal=oryx/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:46.163905 20997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:46.164855 20997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:46.174063 21002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:46.174206 21003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:46.174183 20997 server_base.cc:1034] running on GCE node
W20260619 06:48:46.174063 21005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:46.174731 20997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:46.175309 20997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:46.176458 20997 hybrid_clock.cc:648] HybridClock initialized: now 1781851726176424 us; error 49 us; skew 500 ppm
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:46.179108 20997 init.cc:377] Logged in from keytab as oryx/127.14.116.193@KRBTEST.COM (short username oryx)
I20260619 06:48:46.180114 20997 webserver.cc:460] Webserver started at http://127.14.116.193:36087/ using document root <none> and password file <none>
I20260619 06:48:46.180810 20997 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:46.180902 20997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:46.181142 20997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:46.184428 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "890b226d9d0b460fbc14beb457ba427a"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "925bb149e60dc13b5a53a392f49fffbd"
server_key_iv: "75a38dc26c71bad08789223c0beb5d74"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.185092 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "890b226d9d0b460fbc14beb457ba427a"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "925bb149e60dc13b5a53a392f49fffbd"
server_key_iv: "75a38dc26c71bad08789223c0beb5d74"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.189019 20997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:48:46.192303 21012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.193629 20997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:46.193727 20997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "890b226d9d0b460fbc14beb457ba427a"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "925bb149e60dc13b5a53a392f49fffbd"
server_key_iv: "75a38dc26c71bad08789223c0beb5d74"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.193826 20997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:46.225983 20997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:46.229135 20997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:46.229286 20997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:46.229803 20997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:46.230583 20997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:46.230631 20997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.230692 20997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:46.230716 20997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.239566 20997 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:40389
I20260619 06:48:46.239596 21125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:40389 every 8 connection(s)
I20260619 06:48:46.240451 20997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:46.243032 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 20997
I20260619 06:48:46.243158 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:46.243433 14803 external_mini_cluster.cc:1428] Setting key b8719b63cc27eb11707989b8deb5d597
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.14.116.194@KRBTEST.COM; defaulting to no policy
I20260619 06:48:46.254354 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.242091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:53633 (local address 127.14.116.254:40519)
0619 06:48:46.242218 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.242222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.242993 (+ 771us) server_negotiation.cc:407] Connection header received
0619 06:48:46.244509 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.244513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.244571 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.244667 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.245786 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.246538 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.247698 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.248128 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.248253 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.250596 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.250600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.250605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.250631 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.252355 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.252874 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.252879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.252881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.252964 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.253287 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.253289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.253291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.253467 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.253573 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.254041 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.254159 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.194@KRBTEST.COM" created.
I20260619 06:48:46.255213 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.242415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.193:53633)
0619 06:48:46.242844 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.242884 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:46.244257 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:46.244837 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:46.244844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:46.245178 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:46.245642 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.245654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.246674 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.246676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.247540 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.247549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.248269 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.248272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.248421 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.249059 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:46.249074 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:46.250438 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:46.252473 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.252477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.252487 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.252758 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.253053 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.253058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.253060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.253187 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.253593 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:46.253599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:46.253817 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 06:48:46.254001 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 06:48:46.254312 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":94,"threads_started":1}
I20260619 06:48:46.256412 21126 heartbeater.cc:347] Connected to a master server at 127.14.116.254:40519
I20260619 06:48:46.256667 21126 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:46.257229 21126 heartbeater.cc:511] Master 127.14.116.254:40519 requested a full tablet report, sending...
I20260619 06:48:46.258792 20935 ts_manager.cc:194] Registered new tserver with Master: 890b226d9d0b460fbc14beb457ba427a (127.14.116.193:40389)
I20260619 06:48:46.260478 20935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:53633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab.
Entry for principal oryx/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab.
I20260619 06:48:46.270066 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40519
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab
--principal=oryx/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:46.374545 21131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:46.374773 21131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:46.374812 21131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:46.378682 21131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:46.378736 21131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:46.378790 21131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:46.383738 21131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab
--principal=oryx/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:46.384620 21131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:46.385526 21131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:46.394515 21137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:46.394622 21131 server_base.cc:1034] running on GCE node
W20260619 06:48:46.394515 21139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:46.394591 21136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:46.395154 21131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:46.395756 21131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:46.396922 21131 hybrid_clock.cc:648] HybridClock initialized: now 1781851726396907 us; error 30 us; skew 500 ppm
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:46.399442 21131 init.cc:377] Logged in from keytab as oryx/127.14.116.194@KRBTEST.COM (short username oryx)
I20260619 06:48:46.400337 21131 webserver.cc:460] Webserver started at http://127.14.116.194:34081/ using document root <none> and password file <none>
I20260619 06:48:46.400854 21131 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:46.400939 21131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:46.401170 21131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:46.403141 21131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "ecc0643f08674c3e8fa38ab673b06a96"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "b336edeff88dae69b9431bc08fcfc4ce"
server_key_iv: "302de0183fa2c9b93db26b9806956c6b"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.403762 21131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "ecc0643f08674c3e8fa38ab673b06a96"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "b336edeff88dae69b9431bc08fcfc4ce"
server_key_iv: "302de0183fa2c9b93db26b9806956c6b"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.407912 21131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 06:48:46.410048 21146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.411172 21131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:46.411280 21131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "ecc0643f08674c3e8fa38ab673b06a96"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "b336edeff88dae69b9431bc08fcfc4ce"
server_key_iv: "302de0183fa2c9b93db26b9806956c6b"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.411371 21131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:46.423532 21131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:46.426152 21131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:46.426334 21131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:46.426856 21131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:46.427613 21131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:46.427709 21131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.427768 21131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:46.427816 21131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.436349 21131 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:45885
I20260619 06:48:46.436427 21259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:45885 every 8 connection(s)
I20260619 06:48:46.437242 21131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:46.445559 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 21131
I20260619 06:48:46.445698 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:46.446025 14803 external_mini_cluster.cc:1428] Setting key 991cc7c5d2a78443936931eaa5e5eee4
I20260619 06:48:46.450322 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.438846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:55833 (local address 127.14.116.254:40519)
0619 06:48:46.438992 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.438997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.439860 (+ 863us) server_negotiation.cc:407] Connection header received
0619 06:48:46.441115 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.441118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.441161 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.441234 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.442273 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.442787 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.443882 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.444094 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.444171 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.446640 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.446643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.446645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.446667 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.448318 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.448805 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.448811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.448816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.448908 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.449302 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.449304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.449305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.449468 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.449544 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.450122 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.450210 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:48:46.451107 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.439239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:55833)
0619 06:48:46.439689 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.439742 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:46.440928 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:46.441382 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:46.441388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:46.441715 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:46.442147 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.442159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.442886 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.442888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.443787 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.443795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.444205 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.444208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.444331 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.444816 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:46.444831 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:46.446473 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:46.448454 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.448458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.448468 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.448704 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.449008 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.449014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.449017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.449203 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.449563 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:46.449571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:46.449864 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 06:48:46.450097 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 06:48:46.450352 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260619 06:48:46.452286 21260 heartbeater.cc:347] Connected to a master server at 127.14.116.254:40519
I20260619 06:48:46.452491 21260 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:46.452920 21260 heartbeater.cc:511] Master 127.14.116.254:40519 requested a full tablet report, sending...
I20260619 06:48:46.453970 20935 ts_manager.cc:194] Registered new tserver with Master: ecc0643f08674c3e8fa38ab673b06a96 (127.14.116.194:45885)
I20260619 06:48:46.454521 20935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:55833
WARNING: no policy specified for oryx/127.14.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab.
Entry for principal oryx/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab.
I20260619 06:48:46.472033 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40519
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab
--principal=oryx/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:46.579651 21265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:46.579874 21265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:46.579913 21265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:46.583829 21265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:46.583879 21265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:46.583935 21265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:46.588871 21265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab
--principal=oryx/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:40519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:46.589768 21265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:46.590692 21265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:46.600092 21271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:46.600121 21273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:46.600121 21270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:46.600247 21265 server_base.cc:1034] running on GCE node
I20260619 06:48:46.600677 21265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:46.601243 21265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:46.602396 21265 hybrid_clock.cc:648] HybridClock initialized: now 1781851726602378 us; error 29 us; skew 500 ppm
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:46.605005 21265 init.cc:377] Logged in from keytab as oryx/127.14.116.195@KRBTEST.COM (short username oryx)
I20260619 06:48:46.605835 21265 webserver.cc:460] Webserver started at http://127.14.116.195:46217/ using document root <none> and password file <none>
I20260619 06:48:46.606366 21265 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:46.606451 21265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:46.606664 21265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:46.608656 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "6324d8355f0b4cde941ec554834eef50"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "3f4d3f32b3e6eb3440258683f7bfce3c"
server_key_iv: "52908610651674476894d52dea25d1a3"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.609210 21265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "6324d8355f0b4cde941ec554834eef50"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "3f4d3f32b3e6eb3440258683f7bfce3c"
server_key_iv: "52908610651674476894d52dea25d1a3"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.612663 21265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:48:46.615013 21280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.616158 21265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:46.616289 21265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "6324d8355f0b4cde941ec554834eef50"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "3f4d3f32b3e6eb3440258683f7bfce3c"
server_key_iv: "52908610651674476894d52dea25d1a3"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.616391 21265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:46.626520 21265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:46.629139 21265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:46.629362 21265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:46.629894 21265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:46.630640 21265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:46.630714 21265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.630787 21265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:46.630831 21265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.639281 21265 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:40643
I20260619 06:48:46.639302 21393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:40643 every 8 connection(s)
I20260619 06:48:46.640125 21265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:46.647423 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 21265
I20260619 06:48:46.647542 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:46.647825 14803 external_mini_cluster.cc:1428] Setting key 1567151899ccc11e6a0faca9dd95e416
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:46.652892 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.641740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:35799 (local address 127.14.116.254:40519)
0619 06:48:46.641916 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.641920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.642641 (+ 721us) server_negotiation.cc:407] Connection header received
0619 06:48:46.643976 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.643979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.644034 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.644108 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.645178 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.645730 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.646833 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.647082 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.647209 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.649591 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.649594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.649597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.649627 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.650970 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.651463 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.651468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.651470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.651548 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.651899 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.651902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.651904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.652071 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.652171 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.652623 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.652735 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 06:48:46.653610 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.642097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.195:35799)
0619 06:48:46.642498 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.642537 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:46.643772 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:46.644258 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:46.644265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:46.644580 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:46.645035 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.645047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.645858 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.645862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.646699 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.646707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.647239 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.647243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.647415 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.648057 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:46.648073 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:46.649434 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:46.651097 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.651103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.651116 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.651362 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.651655 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.651660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.651662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.651800 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.652180 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:46.652189 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:46.652410 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 06:48:46.652584 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 06:48:46.652880 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260619 06:48:46.654592 21394 heartbeater.cc:347] Connected to a master server at 127.14.116.254:40519
I20260619 06:48:46.654803 21394 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:46.655232 21394 heartbeater.cc:511] Master 127.14.116.254:40519 requested a full tablet report, sending...
I20260619 06:48:46.656167 20935 ts_manager.cc:194] Registered new tserver with Master: 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643)
I20260619 06:48:46.656680 20935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:35799
I20260619 06:48:46.661187 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.116.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.253.keytab.
Entry for principal oryx/127.14.116.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.253.keytab.
I20260619 06:48:46.684518 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.253:34027
--webserver_interface=127.14.116.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.253.keytab
--principal=oryx/127.14.116.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40519,127.14.116.253:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:46.791823 21399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:46.792058 21399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:46.792098 21399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:46.796190 21399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:46.796250 21399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:46.796283 21399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:46.796305 21399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:46.796325 21399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:46.801656 21399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40519,127.14.116.253:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.253.keytab
--principal=oryx/127.14.116.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.253:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.116.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
W20260619 06:48:46.801826 21399 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.116.254:40519,127.14.116.253:34027'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 06:48:46.802572 21399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:46.803470 21399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:46.811501 21404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:46.811477 21399 server_base.cc:1034] running on GCE node
W20260619 06:48:46.811519 21405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:46.811606 21407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:46.812165 21399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:46.813158 21399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:46.814308 21399 hybrid_clock.cc:648] HybridClock initialized: now 1781851726814296 us; error 24 us; skew 500 ppm
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:46.816846 21399 init.cc:377] Logged in from keytab as oryx/127.14.116.253@KRBTEST.COM (short username oryx)
I20260619 06:48:46.817607 21399 webserver.cc:460] Webserver started at http://127.14.116.253:37789/ using document root <none> and password file <none>
I20260619 06:48:46.818071 21399 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:46.818126 21399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:46.818284 21399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:46.820304 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data/instance:
uuid: "afb9debb2e264c13b53af4b570720a01"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "56ff9150d47d01bb8453bf49557e4216"
server_key_iv: "dad0f776c660071ace7ebef216ea6bee"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.820835 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal/instance:
uuid: "afb9debb2e264c13b53af4b570720a01"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "56ff9150d47d01bb8453bf49557e4216"
server_key_iv: "dad0f776c660071ace7ebef216ea6bee"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.824198 21399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:46.826297 21414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.827392 21399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:46.827481 21399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal
uuid: "afb9debb2e264c13b53af4b570720a01"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "56ff9150d47d01bb8453bf49557e4216"
server_key_iv: "dad0f776c660071ace7ebef216ea6bee"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.827553 21399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:46.840076 21399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:46.843011 21399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:46.843147 21399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:46.849432 21399 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.253:34027
I20260619 06:48:46.849706 21466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.253:34027 every 8 connection(s)
I20260619 06:48:46.850378 21399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
I20260619 06:48:46.852947 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:46.856338 21467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:48:46.859902 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 21399
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.253@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.253@KRBTEST.COM
I20260619 06:48:46.869541 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.857989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.14.116.254:40519)
0619 06:48:46.858109 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.858113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.858791 (+ 678us) server_negotiation.cc:407] Connection header received
0619 06:48:46.859551 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.859553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.859623 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.859727 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.860991 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.861692 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.863125 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.863396 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.863501 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.865950 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.865956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.865962 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.865993 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.867551 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.868070 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.868075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.868077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.868157 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.868486 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.868489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.868491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.868700 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.868799 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.869261 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.869374 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:48:46.870366 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.858270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:37930)
0619 06:48:46.858646 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.858684 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:46.859377 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:46.859874 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:46.859881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:46.860257 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:46.860780 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.860799 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.861822 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.861824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.863009 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.863019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.863524 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.863527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.863721 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.864233 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:46.864254 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:46.865811 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:46.867684 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.867692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.867701 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.867973 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.868249 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.868252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.868254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.868382 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.868807 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:46.868815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:46.869039 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 06:48:46.869210 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 06:48:46.869497 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260619 06:48:46.870942 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.860655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.14.116.253:34027)
0619 06:48:46.860992 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.861000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.861019 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:48:46.861124 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.861128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.861283 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.861408 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.862182 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.863220 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.864146 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.864469 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.864578 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.867359 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.867365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.867380 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.867400 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.868898 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.869315 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.869325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.869327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.869395 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.869697 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.869702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.869706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.869953 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.870091 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.870341 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.870474 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":161,"threads_started":1}
I20260619 06:48:46.871739 21467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.253@KRBTEST.COM for oryx/127.14.116.253@KRBTEST.COM
I20260619 06:48:46.879068 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.872188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.0.0.1:38168)
0619 06:48:46.872274 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.872285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:46.872365 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:46.872567 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:46.872570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:46.872727 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:46.872819 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.872826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.873488 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.873491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.873985 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.873991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.874888 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.874891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.874982 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.875374 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:46.875384 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:46.876536 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:46.877996 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.877999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.878000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.878174 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.878389 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.878391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.878392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.878457 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.878733 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:46.878735 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:46.878808 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 06:48:46.878883 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 06:48:46.878966 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 06:48:46.879180 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.872297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.14.116.253:34027)
0619 06:48:46.872398 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.872402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.872415 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:48:46.872447 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.872450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.872480 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.872560 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.872919 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.873381 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.874086 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.874787 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.874901 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.876644 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.876647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.876648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.876667 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.877903 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.878253 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.878255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.878256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.878319 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.878525 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.878527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.878528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.878663 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.878732 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.878939 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.879018 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260619 06:48:46.881506 21467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: Bootstrap starting.
I20260619 06:48:46.883523 21467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:46.884212 21467 log.cc:824] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:46.885839 21467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: No bootstrap required, opened a new log
I20260619 06:48:46.888651 21467 raft_consensus.cc:348] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } }
I20260619 06:48:46.888823 21467 raft_consensus.cc:374] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:46.888860 21467 raft_consensus.cc:729] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb9debb2e264c13b53af4b570720a01, State: Initialized, Role: FOLLOWER
I20260619 06:48:46.889240 21467 consensus_queue.cc:260] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } }
I20260619 06:48:46.889631 21473 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } } }
I20260619 06:48:46.889761 21473 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:48:46.890285 21467 sys_catalog.cc:564] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:48:46.894526 21484 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:48:46.894611 21484 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:48:46.896522 21399 master_runner.cc:428] Detected that this master afb9debb2e264c13b53af4b570720a01 is joining an existing cluster
I20260619 06:48:46.896569 21399 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.116.253:34027
I20260619 06:48:46.907130 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.898637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.14.116.254:40519)
0619 06:48:46.898761 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.898768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.898988 (+ 220us) server_negotiation.cc:407] Connection header received
0619 06:48:46.899176 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.899179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.899228 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.899330 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.899928 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.900549 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.901729 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.901970 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.902075 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.904062 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.904066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.904069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.904094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.905485 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.905968 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.905972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.905974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.906053 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.906340 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.906343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.906345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.906508 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.906605 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.906859 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.906969 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:48:46.907167 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.898967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.14.116.253:34027)
0619 06:48:46.899075 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.899079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.899323 (+ 244us) server_negotiation.cc:407] Connection header received
0619 06:48:46.899428 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.899431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.899462 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.899505 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.900105 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.900833 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.902065 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.902315 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.902388 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.903650 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.903654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.903657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.903678 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.905495 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.906044 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.906047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.906049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.906116 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.906349 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.906352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.906353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.906520 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.906584 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.906855 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.906998 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":35}
I20260619 06:48:46.907371 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.898542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:37946)
0619 06:48:46.898913 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.898928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:46.899051 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:46.899378 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:46.899381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:46.899634 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:46.899800 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.899808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.900898 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.900902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.901614 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.901622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.902465 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.902467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.902552 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.903524 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:46.903535 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:46.903930 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:46.905595 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.905602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.905604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.905871 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.906142 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.906146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.906148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.906241 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.906606 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:46.906610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:46.906702 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 06:48:46.906805 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:48:46.907092 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":58,"spinlock_wait_cycles":3072,"thread_start_us":90,"threads_started":1}
I20260619 06:48:46.907747 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.898947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.0.0.1:38180)
0619 06:48:46.899239 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.899252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:46.899337 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:46.899589 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:46.899591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:46.899876 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:46.899996 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.900003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.900963 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.900966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.901950 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.901959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.902423 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.902428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.902549 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.903072 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:46.903087 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:46.903484 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:46.905602 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.905605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.905607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.905946 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.906203 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.906205 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.906207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.906272 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.906638 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:46.906640 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:46.906708 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 06:48:46.906917 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 06:48:46.907067 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":66,"spinlock_wait_cycles":3456,"thread_start_us":89,"threads_started":1}
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851725, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.254@KRBTEST.COM for oryx/127.14.116.253@KRBTEST.COM
I20260619 06:48:46.920001 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.911338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.14.116.253:34027)
0619 06:48:46.911454 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.911459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.911804 (+ 345us) server_negotiation.cc:407] Connection header received
0619 06:48:46.911998 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.912000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.912038 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.912095 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.912769 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.913191 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.914349 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.914595 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.914670 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.917026 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.917028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.917030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.917049 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.918381 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.918862 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.918864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.918865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.918972 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.919235 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.919237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.919238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.919399 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.919470 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.919829 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.919908 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:48:46.920220 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.911317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.0.0.1:38188)
0619 06:48:46.911682 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.911733 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:46.911860 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:46.912143 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:46.912149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:46.912546 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:46.912668 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.912676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.913280 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.913283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.914248 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.914258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.914690 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.914693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.914798 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.915231 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:46.915245 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:46.916911 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:46.918477 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.918485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.918490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.918776 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.919051 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.919054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.919056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.919142 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.919493 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:46.919499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:46.919637 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 06:48:46.919788 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 06:48:46.919966 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260619 06:48:46.921593 20935 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.116.253:34027: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "afb9debb2e264c13b53af4b570720a01"
member_type: NON_VOTER
last_known_addr {
host: "127.14.116.253"
port: 34027
}
attrs {
promote: true
}
}
dest_uuid: "a21688a106c343d58a6797563ec5ff64"
cas_config_opid_index: -1
I20260619 06:48:46.922242 20935 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } }
I20260619 06:48:46.923861 21494 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER afb9debb2e264c13b53af4b570720a01 (127.14.116.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } } }
I20260619 06:48:46.924633 21441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:46.925720 21494 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.116.253:34027
I20260619 06:48:46.926081 21399 master.cc:537] Master@127.14.116.253:34027 shutting down...
I20260619 06:48:46.926272 21441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Refusing update from remote peer a21688a106c343d58a6797563ec5ff64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 06:48:46.927153 21496 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:46.928352 21493 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } } }
I20260619 06:48:46.928467 21493 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
W20260619 06:48:46.929251 20921 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 06:48:46.930753 21399 raft_consensus.cc:2229] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:46.931023 21399 raft_consensus.cc:2258] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:46.931125 21399 tablet_replica.cc:330] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: stopping tablet replica
WARNING: no policy specified for oryx/127.14.116.252@KRBTEST.COM; defaulting to no policy
I20260619 06:48:46.935833 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.935447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.0.0.1:38200)
0619 06:48:46.935699 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.935762 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
W20260619 06:48:46.936004 20921 consensus_peers.cc:487] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 -> Peer afb9debb2e264c13b53af4b570720a01 (127.14.116.253:34027): Couldn't send request to peer afb9debb2e264c13b53af4b570720a01. Status: Network error: Client connection negotiation failed: client connection to 127.14.116.253:34027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:48:46.936604 21399 master.cc:552] Master@127.14.116.253:34027 shutdown complete.
I20260619 06:48:46.936651 21399 master_runner.cc:305] Clearing existing system tablet
I20260619 06:48:46.936877 21399 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:46.936941 21399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.252@KRBTEST.COM" created.
I20260619 06:48:46.938153 21399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:46.939036 21502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.939226 21399 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:46.939448 21399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:46.939520 21399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal
uuid: "afb9debb2e264c13b53af4b570720a01"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "56ff9150d47d01bb8453bf49557e4216"
server_key_iv: "dad0f776c660071ace7ebef216ea6bee"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.942126 21399 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:46.944941 21399 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:48:46.945106 21399 log.cc:1196] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 06:48:46.945426 21399 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: Deleting consensus metadata
I20260619 06:48:46.945652 21399 master_runner.cc:315] Copying system tablet from 127.14.116.254:40519
I20260619 06:48:46.946842 21399 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: tablet copy: Beginning tablet copy session from remote peer at address 127.14.116.254:40519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.252.keytab.
Entry for principal oryx/127.14.116.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.252.keytab.
I20260619 06:48:46.952425 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.252:42545
--webserver_interface=127.14.116.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.252.keytab
--principal=oryx/127.14.116.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40519,127.14.116.253:34027,127.14.116.252:42545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:48:46.956017 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.947430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.14.116.254:40519)
0619 06:48:46.947544 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:46.947548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:46.947891 (+ 343us) server_negotiation.cc:407] Connection header received
0619 06:48:46.948082 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:46.948085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:46.948134 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:46.948217 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:46.948728 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.949364 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.950454 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.950691 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.950810 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.952810 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:46.952817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:46.952823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:46.952857 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:46.954457 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.954844 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.954850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.954851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.954932 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:46.955192 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:46.955197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:46.955199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:46.955376 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:46.955475 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:46.955752 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 06:48:46.955866 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:48:46.956029 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:46.947382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:37960)
0619 06:48:46.947789 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:46.947806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:46.947973 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:46.948312 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:46.948317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:46.948480 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:46.948560 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.948565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.949510 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.949517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.950273 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:46.950283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:46.950833 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:46.950838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:46.950934 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:46.951931 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:46.951948 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:46.952390 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:46.954567 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.954569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.954571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.954745 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.955018 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:46.955022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:46.955024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:46.955105 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:46.955475 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:46.955478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:46.955574 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 06:48:46.955701 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 06:48:46.955841 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":80,"threads_started":1}
I20260619 06:48:46.956800 20955 tablet_copy_service.cc:140] P a21688a106c343d58a6797563ec5ff64: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer afb9debb2e264c13b53af4b570720a01 ({username='oryx', principal='oryx/127.14.116.253@KRBTEST.COM'} at 127.0.0.1:37960)
I20260619 06:48:46.956957 20955 tablet_copy_service.cc:161] P a21688a106c343d58a6797563ec5ff64: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer afb9debb2e264c13b53af4b570720a01 at {username='oryx', principal='oryx/127.14.116.253@KRBTEST.COM'} at 127.0.0.1:37960: session id = afb9debb2e264c13b53af4b570720a01-00000000000000000000000000000000
I20260619 06:48:46.958221 20955 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:48:46.959549 21399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:48:46.961244 21399 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: tablet copy: Starting download of 0 data blocks...
I20260619 06:48:46.961424 21399 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: tablet copy: Starting download of 1 WAL segments...
I20260619 06:48:46.962842 21399 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:48:46.964574 20955 tablet_copy_service.cc:342] P a21688a106c343d58a6797563ec5ff64: Request end of tablet copy session afb9debb2e264c13b53af4b570720a01-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.116.253@KRBTEST.COM'} at 127.0.0.1:37960
I20260619 06:48:46.964702 20955 tablet_copy_service.cc:433] P a21688a106c343d58a6797563ec5ff64: ending tablet copy session afb9debb2e264c13b53af4b570720a01-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer afb9debb2e264c13b53af4b570720a01
I20260619 06:48:46.972771 21399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:46.974287 21510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:46.976297 21511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:46.976548 21513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:46.976765 21399 server_base.cc:1034] running on GCE node
I20260619 06:48:46.976907 21399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:46.977165 21399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:46.978302 21399 hybrid_clock.cc:648] HybridClock initialized: now 1781851726978293 us; error 34 us; skew 500 ppm
Jun 19 06:48:46 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851726, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:46.979902 21399 init.cc:377] Logged in from keytab as oryx/127.14.116.253@KRBTEST.COM (short username oryx)
I20260619 06:48:46.980329 21399 webserver.cc:460] Webserver started at http://127.14.116.253:46805/ using document root <none> and password file <none>
I20260619 06:48:46.980508 21399 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:46.980547 21399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:46.981760 21399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:46.982712 21520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:46.982932 21399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:46.983008 21399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal
uuid: "afb9debb2e264c13b53af4b570720a01"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "56ff9150d47d01bb8453bf49557e4216"
server_key_iv: "dad0f776c660071ace7ebef216ea6bee"
server_key_version: "encryptionkey@0"
I20260619 06:48:46.983088 21399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:47.001621 21399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:47.003679 21399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:47.008723 21399 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.253:34027
I20260619 06:48:47.008822 21572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.253:34027 every 8 connection(s)
I20260619 06:48:47.009266 21399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
I20260619 06:48:47.011065 21573 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:48:47.012104 21573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: Bootstrap starting.
I20260619 06:48:47.021973 21573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:48:47.022277 21573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: Bootstrap complete.
I20260619 06:48:47.022850 21573 raft_consensus.cc:348] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } }
I20260619 06:48:47.023075 21573 raft_consensus.cc:729] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: afb9debb2e264c13b53af4b570720a01, State: Initialized, Role: LEARNER
I20260619 06:48:47.023185 21573 consensus_queue.cc:260] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } }
I20260619 06:48:47.023537 21576 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } } }
I20260619 06:48:47.023716 21573 sys_catalog.cc:564] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:47.023736 21576 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:48:47.026779 21587 catalog_manager.cc:1259] Loaded cluster ID: 22e17090ec044d4086624ef9fca979b3
I20260619 06:48:47.026886 21587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: loading cluster ID for follower catalog manager: success
I20260619 06:48:47.028895 21587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: acquiring CA information for follower catalog manager: success
I20260619 06:48:47.030128 21587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:48:47.086709 21509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:47.086937 21509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:47.087025 21509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:47.090921 21509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:47.090971 21509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:47.091015 21509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:47.091049 21509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:47.091080 21509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:47.096098 21509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40519,127.14.116.253:34027,127.14.116.252:42545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.252.keytab
--principal=oryx/127.14.116.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.252:42545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.116.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:47.097019 21509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:47.097961 21509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:47.105888 21592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:47.105888 21593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:47.105895 21595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:47.106220 21509 server_base.cc:1034] running on GCE node
I20260619 06:48:47.106688 21509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:47.107740 21509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:47.108932 21509 hybrid_clock.cc:648] HybridClock initialized: now 1781851727108908 us; error 31 us; skew 500 ppm
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:47.111460 21509 init.cc:377] Logged in from keytab as oryx/127.14.116.252@KRBTEST.COM (short username oryx)
I20260619 06:48:47.112309 21509 webserver.cc:460] Webserver started at http://127.14.116.252:42107/ using document root <none> and password file <none>
I20260619 06:48:47.112830 21509 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:47.112888 21509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:47.113121 21509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:47.115053 21509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data/instance:
uuid: "2b5a2dd1ea1040b09762a53247d70e05"
format_stamp: "Formatted at 2026-06-19 06:48:47 on dist-test-slave-9x3b"
server_key: "1d95538edefb85d721ed8d44ad0efca6"
server_key_iv: "f171145562b1dc3ef7f26f0349b232ca"
server_key_version: "encryptionkey@0"
I20260619 06:48:47.115643 21509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal/instance:
uuid: "2b5a2dd1ea1040b09762a53247d70e05"
format_stamp: "Formatted at 2026-06-19 06:48:47 on dist-test-slave-9x3b"
server_key: "1d95538edefb85d721ed8d44ad0efca6"
server_key_iv: "f171145562b1dc3ef7f26f0349b232ca"
server_key_version: "encryptionkey@0"
I20260619 06:48:47.119076 21509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:48:47.121490 21602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.122635 21509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:47.122752 21509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal
uuid: "2b5a2dd1ea1040b09762a53247d70e05"
format_stamp: "Formatted at 2026-06-19 06:48:47 on dist-test-slave-9x3b"
server_key: "1d95538edefb85d721ed8d44ad0efca6"
server_key_iv: "f171145562b1dc3ef7f26f0349b232ca"
server_key_version: "encryptionkey@0"
I20260619 06:48:47.122852 21509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:47.145346 21509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:47.148381 21509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:47.148568 21509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:47.155782 21509 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.252:42545
I20260619 06:48:47.155954 21654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.252:42545 every 8 connection(s)
I20260619 06:48:47.156744 21509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
I20260619 06:48:47.159312 21655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:47.162989 21655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:48:47.166657 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 21509
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.252@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.252@KRBTEST.COM
I20260619 06:48:47.176532 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.164829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.14.116.254:40519)
0619 06:48:47.164957 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.164961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.165681 (+ 720us) server_negotiation.cc:407] Connection header received
0619 06:48:47.166613 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.166616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.166659 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.166750 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.168085 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.168707 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.169853 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.170062 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.170141 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.172798 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.172801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.172803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.172824 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.174372 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.175036 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.175038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.175039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.175110 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.175462 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.175464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.175465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.175667 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.175774 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.176235 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.176368 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:48:47.177352 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.167471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.14.116.252:42545)
0619 06:48:47.167840 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.167849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.167866 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:48:47.167899 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.167902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.168059 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.168214 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.168854 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.169700 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.170734 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.171059 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.171167 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.173690 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.173696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.173704 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.173723 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.175452 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.175883 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.175889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.175890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.175978 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.176237 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.176240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.176242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.176460 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.176653 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.176828 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.176960 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":258,"spinlock_wait_cycles":2816,"thread_start_us":178,"threads_started":1}
I20260619 06:48:47.177354 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.165135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:37964)
0619 06:48:47.165542 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.165581 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.166390 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.166934 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.166943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.167338 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.167953 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.167966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.168850 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.168853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.169718 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.169726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.170196 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.170201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.170382 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.170987 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.171002 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.172662 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.174494 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.174506 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.174516 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.174945 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.175198 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.175202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.175204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.175364 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.175772 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.175788 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.176016 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.176193 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.176471 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":45,"spinlock_wait_cycles":2304,"thread_start_us":94,"threads_started":1}
I20260619 06:48:47.178520 21655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.252@KRBTEST.COM for oryx/127.14.116.253@KRBTEST.COM
I20260619 06:48:47.186054 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.178956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.0.0.1:38212)
0619 06:48:47.179095 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.179110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.179177 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.179561 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.179563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.179733 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.179812 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.179816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.180647 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.180649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.181154 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.181161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.181540 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.181542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.181608 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.181958 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.181967 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.183038 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.184686 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.184690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.184692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.184880 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.185149 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.185152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.185154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.185233 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.185597 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.185601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.185688 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.185789 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.185905 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 06:48:47.186105 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.179062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.14.116.253:34027)
0619 06:48:47.179377 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.179381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.179392 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:48:47.179424 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.179426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.179463 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.179532 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.179918 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.180562 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.181244 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.181457 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.181531 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.183183 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.183186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.183188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.183210 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.184587 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.184973 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.184978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.184980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.185057 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.185332 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.185336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.185337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.185500 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.185596 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.185841 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.185953 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
I20260619 06:48:47.186730 21655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.252@KRBTEST.COM for oryx/127.14.116.252@KRBTEST.COM
I20260619 06:48:47.193346 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.187078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.0.0.1:51184)
0619 06:48:47.187175 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.187189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.187254 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.187421 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.187423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.187537 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.187595 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.187599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.188309 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.188315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.188787 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.188793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.189204 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.189208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.189277 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.189673 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.189685 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.190718 (+ 1033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.192101 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.192103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.192104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.192273 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.192486 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.192487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.192489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.192553 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.192872 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.192875 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.192950 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.193054 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.193178 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 06:48:47.193354 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.187147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.14.116.252:42545)
0619 06:48:47.187247 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.187250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.187263 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:48:47.187303 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.187305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.187333 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.187394 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.187728 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.188201 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.188888 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.189110 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.189208 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.190837 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.190840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.190842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.190861 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.192013 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.192350 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.192352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.192353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.192416 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.192620 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.192622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.192623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.192757 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.192881 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.193107 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.193232 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":39}
I20260619 06:48:47.197003 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Bootstrap starting.
I20260619 06:48:47.199200 21655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:47.199956 21655 log.cc:824] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:47.201694 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: No bootstrap required, opened a new log
I20260619 06:48:47.204572 21655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } }
I20260619 06:48:47.204751 21655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:47.204782 21655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5a2dd1ea1040b09762a53247d70e05, State: Initialized, Role: FOLLOWER
I20260619 06:48:47.205171 21655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } }
I20260619 06:48:47.205600 21661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } } }
I20260619 06:48:47.205756 21661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:48:47.206249 21655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:48:47.210696 21672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:48:47.210775 21672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:48:47.213842 21509 master_runner.cc:428] Detected that this master 2b5a2dd1ea1040b09762a53247d70e05 is joining an existing cluster
I20260619 06:48:47.213900 21509 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.116.252:42545
I20260619 06:48:47.224465 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.216173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.14.116.252:42545)
0619 06:48:47.216272 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.216275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.216847 (+ 572us) server_negotiation.cc:407] Connection header received
0619 06:48:47.217304 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.217307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.217345 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.217389 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.217910 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.218390 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.219679 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.220020 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.220099 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.221489 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.221493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.221496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.221516 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.222861 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.223314 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.223320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.223322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.223390 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.223679 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.223685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.223687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.223836 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.223969 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.224222 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.224313 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":39}
I20260619 06:48:47.224478 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.216074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.0.0.1:51200)
0619 06:48:47.216784 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.216795 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.216883 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.217504 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.217509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.217720 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.217806 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.217811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.218487 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.218493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.219529 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.219539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.220277 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.220280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.220405 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.220965 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.220984 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.221342 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.222988 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.222992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.222994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.223229 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.223474 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.223479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.223481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.223558 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.223959 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.223962 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.224044 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.224150 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.224271 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":158,"thread_start_us":93,"threads_started":1}
I20260619 06:48:47.230258 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 20997
I20260619 06:48:47.236956 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:40389
--local_ip_for_outbound_sockets=127.14.116.193
--tserver_master_addrs=127.14.116.254:40519
--webserver_port=36087
--webserver_interface=127.14.116.193
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab
--principal=oryx/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.116.254:40519,127.14.116.253:34027
--tserver_master_addrs=127.14.116.254:40519,127.14.116.253:34027,127.14.116.252:42545 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:48:47.248309 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.216544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.14.116.253:34027)
0619 06:48:47.216653 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.216657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.217131 (+ 474us) server_negotiation.cc:407] Connection header received
0619 06:48:47.217287 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.217290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.217337 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.217412 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.219012 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.219835 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.221039 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.221411 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.221483 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.244893 (+ 23410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.244897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.244900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.244929 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.246496 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.246991 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.246995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.246997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.247069 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.247345 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.247348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.247350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.247565 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.247668 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.248013 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.248143 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:48:47.248805 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.216068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:37972)
0619 06:48:47.216711 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.216725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.216831 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.217094 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.217100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.217345 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.217479 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.217487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.218523 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.218526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.219550 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.219556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.220277 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.220280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.220448 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.244092 (+ 23644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.244113 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.244686 (+ 573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.246943 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.246949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.246951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.247215 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.247559 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.247563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.247568 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.247683 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.248279 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.248284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.248373 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.248472 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.248625 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":131,"thread_start_us":91,"threads_started":1}
I20260619 06:48:47.248900 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.216488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.0.0.1:38218)
0619 06:48:47.217059 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.217071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.217187 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.217519 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.217522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.218742 (+ 1220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.218878 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.218885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.220003 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.220007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.220906 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.220915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.221634 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.221637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.221724 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.244232 (+ 22508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.244248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.244703 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.246637 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.246641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.246643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.246880 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.247170 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.247173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.247175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.247251 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.247737 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.247740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.247844 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.248490 (+ 646us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.248693 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":53,"thread_start_us":79,"threads_started":1}
I20260619 06:48:47.249686 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.216097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.14.116.254:40519)
0619 06:48:47.216232 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.216237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.216801 (+ 564us) server_negotiation.cc:407] Connection header received
0619 06:48:47.216918 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.216922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.216971 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.217062 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.217610 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.218438 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.219713 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.220052 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.220121 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.244913 (+ 24792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.244918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.244922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.244963 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.246783 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.247341 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.247346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.247351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.247442 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.247818 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.247822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.247823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.248122 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.248229 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.249417 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.249536 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851725, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.254@KRBTEST.COM for oryx/127.14.116.252@KRBTEST.COM
I20260619 06:48:47.262161 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.253390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.0.0.1:51202)
0619 06:48:47.253545 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.253559 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.253625 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.253877 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.253880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.254053 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.254148 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.254156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.255199 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.255203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.256210 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.256220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.256777 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.256781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.256878 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.257333 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.257348 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.258794 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.260721 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.260724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.260726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.260962 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.261236 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.261239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.261241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.261323 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.261678 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.261681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.261778 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.261883 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.262007 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 06:48:47.262182 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.253475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.14.116.252:42545)
0619 06:48:47.253575 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.253580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.253592 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:48:47.253726 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.253729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.253775 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.253852 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.254283 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.255091 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.256326 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.256662 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.256762 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.258945 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.258949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.258952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.258975 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.260619 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.261057 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.261060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.261062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.261140 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.261407 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.261410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.261412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.261588 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.261678 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.261951 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.262051 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:48:47.263268 20935 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.116.252:42545: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05"
member_type: NON_VOTER
last_known_addr {
host: "127.14.116.252"
port: 42545
}
attrs {
promote: true
}
}
dest_uuid: "a21688a106c343d58a6797563ec5ff64"
cas_config_opid_index: 5
I20260619 06:48:47.263867 20935 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } }
I20260619 06:48:47.265825 21496 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2b5a2dd1ea1040b09762a53247d70e05 (127.14.116.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.267298 21496 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.116.252:42545
I20260619 06:48:47.267445 21497 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.267549 21497 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:47.268025 21629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:47.269304 21629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Refusing update from remote peer a21688a106c343d58a6797563ec5ff64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 06:48:47.269851 21497 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:47.270467 21509 master.cc:537] Master@127.14.116.252:42545 shutting down...
W20260619 06:48:47.271938 20920 consensus_peers.cc:487] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 -> Peer 2b5a2dd1ea1040b09762a53247d70e05 (127.14.116.252:42545): Couldn't send request to peer 2b5a2dd1ea1040b09762a53247d70e05. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 06:48:47.274936 21509 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:47.275189 21509 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:47.275285 21509 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: stopping tablet replica
I20260619 06:48:47.276966 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.269742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.0.0.1:38220)
0619 06:48:47.271956 (+ 2214us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.271971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.272274 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.272649 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.272652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.272667 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:47.272744 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.272749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.273875 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.273878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.275207 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.275217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.276543 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.276547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.276649 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.276694 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.276773 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.276840 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2143}
I20260619 06:48:47.277458 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.269898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.14.116.253:34027)
0619 06:48:47.271519 (+ 1621us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.271523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.272207 (+ 684us) server_negotiation.cc:407] Connection header received
0619 06:48:47.272391 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.272394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.272452 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.272513 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:47.272873 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.273636 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.275433 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.276115 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.276202 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.276326 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.277270 (+ 944us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.277350 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1540}
I20260619 06:48:47.278178 21547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 LEARNER]: Refusing update from remote peer a21688a106c343d58a6797563ec5ff64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 06:48:47.278858 21497 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 06:48:47.280723 21547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2b5a2dd1ea1040b09762a53247d70e05 (127.14.116.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.282497 21683 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.282647 21683 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:48:47.282909 21576 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21688a106c343d58a6797563ec5ff64. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: NON_VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: true } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.283015 21576 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:48:47.283172 21509 master.cc:552] Master@127.14.116.252:42545 shutdown complete.
I20260619 06:48:47.283213 21509 master_runner.cc:305] Clearing existing system tablet
I20260619 06:48:47.283501 21509 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:47.283563 21509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:47.284639 21497 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: attempting to promote NON_VOTER afb9debb2e264c13b53af4b570720a01 to VOTER
I20260619 06:48:47.285075 21497 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } }
I20260619 06:48:47.286453 21547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 LEARNER]: Refusing update from remote peer a21688a106c343d58a6797563ec5ff64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 06:48:47.286864 21497 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 06:48:47.287979 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.285925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.0.0.1:51208)
0619 06:48:47.287844 (+ 1919us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.287906 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1841}
I20260619 06:48:47.288650 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.288429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.0.0.1:51220)
0619 06:48:47.288548 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.288591 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 06:48:47.288757 20920 consensus_peers.cc:487] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 -> Peer 2b5a2dd1ea1040b09762a53247d70e05 (127.14.116.252:42545): Couldn't send request to peer 2b5a2dd1ea1040b09762a53247d70e05. Status: Network error: Client connection negotiation failed: client connection to 127.14.116.252:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:48:47.288866 21497 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, afb9debb2e264c13b53af4b570720a01 (127.14.116.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.290266 21497 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.290362 21497 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:47.290477 21497 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.290562 21497 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:47.291338 21547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, afb9debb2e264c13b53af4b570720a01 (127.14.116.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.292709 21576 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.292820 21576 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:48:47.293217 21509 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.000s sys 0.001s
I20260619 06:48:47.294114 21685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.294315 21509 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:47.294584 21509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 06:48:47.294662 21509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal
uuid: "2b5a2dd1ea1040b09762a53247d70e05"
format_stamp: "Formatted at 2026-06-19 06:48:47 on dist-test-slave-9x3b"
server_key: "1d95538edefb85d721ed8d44ad0efca6"
server_key_iv: "f171145562b1dc3ef7f26f0349b232ca"
server_key_version: "encryptionkey@0"
I20260619 06:48:47.296808 21509 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:47.299278 21509 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:48:47.299423 21509 log.cc:1196] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 06:48:47.299787 21509 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Deleting consensus metadata
I20260619 06:48:47.300027 21509 master_runner.cc:315] Copying system tablet from 127.14.116.254:40519
I20260619 06:48:47.301258 21509 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: tablet copy: Beginning tablet copy session from remote peer at address 127.14.116.254:40519
I20260619 06:48:47.310364 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.302099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.14.116.254:40519)
0619 06:48:47.302212 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.302217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.302428 (+ 211us) server_negotiation.cc:407] Connection header received
0619 06:48:47.302603 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.302606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.302651 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.302700 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.303297 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.304115 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.305275 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.305599 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.305703 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.306972 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.306976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.306979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.307005 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.308666 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.309133 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.309136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.309138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.309223 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.309514 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.309517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.309519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.309728 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.309822 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.310076 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.310184 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:48:47.310354 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.301780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:37978)
0619 06:48:47.302346 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.302360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.302499 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.302848 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.302851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.303058 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.303166 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.303172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.304241 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.304244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.305152 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.305162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.305718 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.305722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.305816 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.306394 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.306407 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.306833 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.308781 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.308784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.308787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.309034 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.309312 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.309315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.309317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.309413 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.309818 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.309821 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.309910 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.310013 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.310137 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":97,"threads_started":1}
I20260619 06:48:47.311194 20955 tablet_copy_service.cc:140] P a21688a106c343d58a6797563ec5ff64: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2b5a2dd1ea1040b09762a53247d70e05 ({username='oryx', principal='oryx/127.14.116.252@KRBTEST.COM'} at 127.0.0.1:37978)
I20260619 06:48:47.311287 20955 tablet_copy_service.cc:161] P a21688a106c343d58a6797563ec5ff64: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2b5a2dd1ea1040b09762a53247d70e05 at {username='oryx', principal='oryx/127.14.116.252@KRBTEST.COM'} at 127.0.0.1:37978: session id = 2b5a2dd1ea1040b09762a53247d70e05-00000000000000000000000000000000
I20260619 06:48:47.312327 20955 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:48:47.313596 21509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:48:47.315790 21509 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: tablet copy: Starting download of 0 data blocks...
I20260619 06:48:47.315961 21509 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: tablet copy: Starting download of 1 WAL segments...
I20260619 06:48:47.317237 21509 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:48:47.318846 20955 tablet_copy_service.cc:342] P a21688a106c343d58a6797563ec5ff64: Request end of tablet copy session 2b5a2dd1ea1040b09762a53247d70e05-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.116.252@KRBTEST.COM'} at 127.0.0.1:37978
I20260619 06:48:47.318919 20955 tablet_copy_service.cc:433] P a21688a106c343d58a6797563ec5ff64: ending tablet copy session 2b5a2dd1ea1040b09762a53247d70e05-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2b5a2dd1ea1040b09762a53247d70e05
I20260619 06:48:47.326025 21509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:47.327687 21692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:47.332501 21693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:47.332849 21509 server_base.cc:1034] running on GCE node
W20260619 06:48:47.333042 21695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:47.334501 21509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:47.334865 21509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:47.336024 21509 hybrid_clock.cc:648] HybridClock initialized: now 1781851727336012 us; error 31 us; skew 500 ppm
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:47.337626 21509 init.cc:377] Logged in from keytab as oryx/127.14.116.252@KRBTEST.COM (short username oryx)
I20260619 06:48:47.338088 21509 webserver.cc:460] Webserver started at http://127.14.116.252:36839/ using document root <none> and password file <none>
I20260619 06:48:47.338300 21509 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:47.338366 21509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:47.339721 21509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:47.340734 21702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.340958 21509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:47.341065 21509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal
uuid: "2b5a2dd1ea1040b09762a53247d70e05"
format_stamp: "Formatted at 2026-06-19 06:48:47 on dist-test-slave-9x3b"
server_key: "1d95538edefb85d721ed8d44ad0efca6"
server_key_iv: "f171145562b1dc3ef7f26f0349b232ca"
server_key_version: "encryptionkey@0"
I20260619 06:48:47.341207 21509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:47.358047 21509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:47.360420 21509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:47.365500 21509 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.252:42545
I20260619 06:48:47.365581 21757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.252:42545 every 8 connection(s)
I20260619 06:48:47.366290 21509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
I20260619 06:48:47.368132 21758 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:48:47.369140 21758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Bootstrap starting.
W20260619 06:48:47.369472 21681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:47.369773 21681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:47.369848 21681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:47.375636 21681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:47.375705 21681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:47.375785 21681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:47.380026 21758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:48:47.380316 21758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Bootstrap complete.
I20260619 06:48:47.380846 21758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } }
I20260619 06:48:47.380995 21758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2b5a2dd1ea1040b09762a53247d70e05, State: Initialized, Role: LEARNER
I20260619 06:48:47.381098 21758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } }
I20260619 06:48:47.381454 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.381630 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:48:47.381548 21758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:47.383066 21681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab
--principal=oryx/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:40389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=36087
--tserver_master_addrs=127.14.116.254:40519,127.14.116.253:34027,127.14.116.252:42545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:47.384508 21681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:47.384878 21772 catalog_manager.cc:1259] Loaded cluster ID: 22e17090ec044d4086624ef9fca979b3
I20260619 06:48:47.384918 21772 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: loading cluster ID for follower catalog manager: success
I20260619 06:48:47.385641 21681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:48:47.387171 21772 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: acquiring CA information for follower catalog manager: success
I20260619 06:48:47.388445 21772 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:48:47.394840 21775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:47.394856 21774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:47.395078 21681 server_base.cc:1034] running on GCE node
W20260619 06:48:47.395282 21777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:47.395745 21681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:47.396420 21681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:47.397591 21681 hybrid_clock.cc:648] HybridClock initialized: now 1781851727397570 us; error 31 us; skew 500 ppm
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:47.400276 21681 init.cc:377] Logged in from keytab as oryx/127.14.116.193@KRBTEST.COM (short username oryx)
I20260619 06:48:47.401131 21681 webserver.cc:460] Webserver started at http://127.14.116.193:36087/ using document root <none> and password file <none>
I20260619 06:48:47.401685 21681 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:47.401782 21681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:47.405494 21681 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 06:48:47.407521 21784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.408766 21681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:47.408890 21681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "890b226d9d0b460fbc14beb457ba427a"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "925bb149e60dc13b5a53a392f49fffbd"
server_key_iv: "75a38dc26c71bad08789223c0beb5d74"
server_key_version: "encryptionkey@0"
I20260619 06:48:47.409425 21681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:47.429340 21681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:47.431984 21681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:47.432168 21681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:47.432713 21681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:47.433517 21681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:47.433583 21681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.433646 21681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:47.433691 21681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.443472 21681 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:40389
I20260619 06:48:47.443497 21897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:40389 every 8 connection(s)
I20260619 06:48:47.444411 21681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:47.447785 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 21681
I20260619 06:48:47.447978 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 21131
I20260619 06:48:47.456199 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:45885
--local_ip_for_outbound_sockets=127.14.116.194
--tserver_master_addrs=127.14.116.254:40519
--webserver_port=34081
--webserver_interface=127.14.116.194
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab
--principal=oryx/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.116.254:40519,127.14.116.253:34027
--tserver_master_addrs=127.14.116.254:40519,127.14.116.253:34027,127.14.116.252:42545 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.253@KRBTEST.COM
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.252@KRBTEST.COM
I20260619 06:48:47.462949 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.447518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:54667 (local address 127.14.116.253:34027)
0619 06:48:47.447663 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.447669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.447922 (+ 253us) server_negotiation.cc:407] Connection header received
0619 06:48:47.448891 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.448905 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.448963 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.449037 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.450724 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.451528 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.453716 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.454057 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.454129 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.458525 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.458530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.458533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.458563 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.460473 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.461174 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.461178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.461179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.461260 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.461678 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.461682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.461684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.461889 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.461974 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.462672 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.462786 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:48:47.463979 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.446433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.14.116.193:40921)
0619 06:48:47.447376 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.447389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.448443 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.449133 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.449142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.449784 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.450565 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.450582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.451922 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.451925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.453582 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.453592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.454327 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.454331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.454801 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.455724 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.455738 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.458844 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.460793 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.460796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.460798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.461082 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.461361 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.461363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.461365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.461569 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.462032 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.462038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.462384 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.462608 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.462931 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":699,"spinlock_wait_cycles":14464,"thread_start_us":135,"threads_started":1}
I20260619 06:48:47.464170 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.446035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:42623 (local address 127.14.116.254:40519)
0619 06:48:47.446180 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.446185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.447059 (+ 874us) server_negotiation.cc:407] Connection header received
0619 06:48:47.449075 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.449081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.449144 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.449201 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.450728 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.451694 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.453702 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.454124 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.454221 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.459797 (+ 5576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.459804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.459810 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.460062 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.461923 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.462507 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.462514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.462519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.462613 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.462972 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.462978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.462983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.463238 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.463334 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.463809 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.463943 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:48:47.464461 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.447409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.14.116.193:54667)
0619 06:48:47.447781 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.447791 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.448769 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.449133 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.449142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.449784 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.450566 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.450582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.451717 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.451720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.453583 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.453592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.454327 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.454331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.454795 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.455673 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.455698 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.458363 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.460611 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.460619 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.460635 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.461060 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.461363 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.461365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.461367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.461569 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.462032 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.462038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.462384 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.463162 (+ 778us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.463287 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":650,"spinlock_wait_cycles":25472,"thread_start_us":80,"threads_started":1}
I20260619 06:48:47.464787 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.446431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.193:42623)
0619 06:48:47.446902 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.446950 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.448013 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.449394 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.449399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.449785 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.450560 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.450581 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.451837 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.451843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.453565 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.453578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.454432 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.454438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.454838 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.456004 (+ 1166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.456019 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.459125 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.462075 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.462080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.462085 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.462376 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.462746 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.462752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.462755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.462853 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.463422 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.463427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.463540 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.464351 (+ 811us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.464527 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":548,"spinlock_wait_cycles":29696,"thread_start_us":533,"threads_started":1}
I20260619 06:48:47.465576 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.446169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:40921 (local address 127.14.116.252:42545)
0619 06:48:47.446430 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.446443 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.447456 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 06:48:47.448652 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.448658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.448722 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.448791 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.450724 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.451796 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.453716 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.454057 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.454129 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.458983 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.458987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.458989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.459018 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.460683 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.461185 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.461188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.461190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.461260 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.461678 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.461682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.461684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.461901 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.461980 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.465313 (+ 3333us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.465463 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260619 06:48:47.465875 21899 heartbeater.cc:347] Connected to a master server at 127.14.116.254:40519
I20260619 06:48:47.466151 21899 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:47.466315 21900 heartbeater.cc:347] Connected to a master server at 127.14.116.253:34027
I20260619 06:48:47.466411 21900 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:47.466764 21900 heartbeater.cc:511] Master 127.14.116.253:34027 requested a full tablet report, sending...
I20260619 06:48:47.466768 21899 heartbeater.cc:511] Master 127.14.116.254:40519 requested a full tablet report, sending...
I20260619 06:48:47.466812 21898 heartbeater.cc:347] Connected to a master server at 127.14.116.252:42545
I20260619 06:48:47.466938 21898 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:47.467106 21898 heartbeater.cc:511] Master 127.14.116.252:42545 requested a full tablet report, sending...
I20260619 06:48:47.468127 20935 ts_manager.cc:194] Re-registered known tserver with Master: 890b226d9d0b460fbc14beb457ba427a (127.14.116.193:40389)
I20260619 06:48:47.468329 21529 ts_manager.cc:194] Registered new tserver with Master: 890b226d9d0b460fbc14beb457ba427a (127.14.116.193:40389)
I20260619 06:48:47.468328 21719 ts_manager.cc:194] Registered new tserver with Master: 890b226d9d0b460fbc14beb457ba427a (127.14.116.193:40389)
I20260619 06:48:47.468969 20935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:42623
W20260619 06:48:47.567334 21906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:47.567574 21906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:47.567667 21906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:47.571524 21906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:47.571578 21906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:47.571650 21906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:47.576690 21906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab
--principal=oryx/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:45885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=34081
--tserver_master_addrs=127.14.116.254:40519,127.14.116.253:34027,127.14.116.252:42545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:47.577626 21906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:47.578553 21906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:47.588233 21911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:47.588285 21912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:47.588320 21906 server_base.cc:1034] running on GCE node
W20260619 06:48:47.588233 21914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:47.588778 21906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:47.589419 21906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:47.590610 21906 hybrid_clock.cc:648] HybridClock initialized: now 1781851727590589 us; error 31 us; skew 500 ppm
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:47.593402 21906 init.cc:377] Logged in from keytab as oryx/127.14.116.194@KRBTEST.COM (short username oryx)
I20260619 06:48:47.594266 21906 webserver.cc:460] Webserver started at http://127.14.116.194:34081/ using document root <none> and password file <none>
I20260619 06:48:47.594820 21906 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:47.594924 21906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:47.598520 21906 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:47.600477 21921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.601711 21906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:47.601835 21906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "ecc0643f08674c3e8fa38ab673b06a96"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "b336edeff88dae69b9431bc08fcfc4ce"
server_key_iv: "302de0183fa2c9b93db26b9806956c6b"
server_key_version: "encryptionkey@0"
I20260619 06:48:47.602342 21906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:47.621294 21906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:47.624089 21906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:47.624280 21906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:47.624881 21906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:47.625833 21906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:47.625893 21906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.625957 21906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:47.625988 21906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.634716 21906 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:45885
I20260619 06:48:47.634747 22034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:45885 every 8 connection(s)
I20260619 06:48:47.635653 21906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:47.643144 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 21906
I20260619 06:48:47.643325 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 21265
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.252@KRBTEST.COM
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.253@KRBTEST.COM
I20260619 06:48:47.650357 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:40643
--local_ip_for_outbound_sockets=127.14.116.195
--tserver_master_addrs=127.14.116.254:40519
--webserver_port=46217
--webserver_interface=127.14.116.195
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab
--principal=oryx/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.116.254:40519,127.14.116.253:34027
--tserver_master_addrs=127.14.116.254:40519,127.14.116.253:34027,127.14.116.252:42545 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:48:47.652038 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.638977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:55631 (local address 127.14.116.253:34027)
0619 06:48:47.639098 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.639102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.639207 (+ 105us) server_negotiation.cc:407] Connection header received
0619 06:48:47.639633 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.639637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.639697 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.639752 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.641581 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.642387 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.644985 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.645312 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.645371 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.648282 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.648288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.648294 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.648333 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.650044 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.650554 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.650560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.650564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.650645 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.650958 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.650964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.650969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.651204 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.651278 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.651694 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.651819 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:47.652881 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.638854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.14.116.194:55631)
0619 06:48:47.639135 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.639147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.639395 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.640068 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.640075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.640688 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.641441 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.641451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.642583 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.642586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.644864 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.644872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.645553 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.645557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.645648 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.646085 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.646100 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.648097 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.650164 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.650171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.650173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.650437 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.650759 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.650765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.650770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.650861 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.651363 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.651368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.651478 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.652478 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.652638 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":206,"spinlock_wait_cycles":132992,"thread_start_us":87,"threads_started":1}
I20260619 06:48:47.653144 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.637749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.14.116.194:33105)
0619 06:48:47.638636 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.638649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.639384 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.640066 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.640075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.640679 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.641411 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.641429 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.642571 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.642574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.643993 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.644002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.644701 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.644705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.644939 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.645586 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.645605 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.647466 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.649485 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.649494 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.649511 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.649888 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.650193 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.650196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.650198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.650384 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.650832 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.650838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.651142 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.651339 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.651736 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":226,"spinlock_wait_cycles":32384,"thread_start_us":159,"threads_started":1}
I20260619 06:48:47.653549 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.637442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:33105 (local address 127.14.116.252:42545)
0619 06:48:47.637569 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.637575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.638723 (+ 1148us) server_negotiation.cc:407] Connection header received
0619 06:48:47.639708 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.639711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.639767 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.639835 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.641574 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.642395 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.644107 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.644460 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.644556 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.647636 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.647641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.647644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.647668 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.649370 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.650013 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.650016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.650018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.650101 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.650488 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.650492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.650494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.650707 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.650786 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.653260 (+ 2474us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.653388 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:47.654366 22035 heartbeater.cc:347] Connected to a master server at 127.14.116.252:42545
I20260619 06:48:47.654445 22037 heartbeater.cc:347] Connected to a master server at 127.14.116.253:34027
I20260619 06:48:47.654569 22035 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:47.654569 22037 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:47.655190 22037 heartbeater.cc:511] Master 127.14.116.253:34027 requested a full tablet report, sending...
I20260619 06:48:47.655190 22035 heartbeater.cc:511] Master 127.14.116.252:42545 requested a full tablet report, sending...
I20260619 06:48:47.656304 21719 ts_manager.cc:194] Registered new tserver with Master: ecc0643f08674c3e8fa38ab673b06a96 (127.14.116.194:45885)
I20260619 06:48:47.656378 21529 ts_manager.cc:194] Registered new tserver with Master: ecc0643f08674c3e8fa38ab673b06a96 (127.14.116.194:45885)
I20260619 06:48:47.658210 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.637274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:54633 (local address 127.14.116.254:40519)
0619 06:48:47.637579 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.637585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.638382 (+ 797us) server_negotiation.cc:407] Connection header received
0619 06:48:47.639634 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.639638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.639696 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.639752 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.641593 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.642403 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.650619 (+ 8216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.650950 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.651027 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.653330 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.653333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.653336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.653366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.655109 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.657058 (+ 1949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.657060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.657062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.657136 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.657473 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.657475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.657476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.657634 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.657704 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.658016 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.658106 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260619 06:48:47.658375 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.637749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:54633)
0619 06:48:47.638241 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.638279 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.639384 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.640066 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.640075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.640679 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.641411 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.641429 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.642571 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.642574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.650493 (+ 7919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.650501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.651181 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.651184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.651268 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.651756 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.651769 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.653173 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.655216 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.655218 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.655220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.656965 (+ 1745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.657230 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.657235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.657239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.657378 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.657721 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.657726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.657837 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.657960 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.658116 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":175,"thread_start_us":648,"threads_started":1}
I20260619 06:48:47.658917 22036 heartbeater.cc:347] Connected to a master server at 127.14.116.254:40519
I20260619 06:48:47.659008 22036 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:47.659164 22036 heartbeater.cc:511] Master 127.14.116.254:40519 requested a full tablet report, sending...
I20260619 06:48:47.659801 20935 ts_manager.cc:194] Re-registered known tserver with Master: ecc0643f08674c3e8fa38ab673b06a96 (127.14.116.194:45885)
I20260619 06:48:47.660362 20935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:54633
W20260619 06:48:47.762522 22042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:47.762740 22042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:47.762779 22042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:47.766923 22042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:47.766996 22042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:47.767093 22042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:47.772063 22042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab
--principal=oryx/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:40643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=46217
--tserver_master_addrs=127.14.116.254:40519,127.14.116.253:34027,127.14.116.252:42545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:47.772962 22042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:47.773897 22042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:47.783258 22047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:47.783263 22048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:47.783247 22050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:47.783589 22042 server_base.cc:1034] running on GCE node
I20260619 06:48:47.784039 22042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:47.784686 22042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:47.785928 22042 hybrid_clock.cc:648] HybridClock initialized: now 1781851727785902 us; error 36 us; skew 500 ppm
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:47.788641 22042 init.cc:377] Logged in from keytab as oryx/127.14.116.195@KRBTEST.COM (short username oryx)
I20260619 06:48:47.789520 22042 webserver.cc:460] Webserver started at http://127.14.116.195:46217/ using document root <none> and password file <none>
I20260619 06:48:47.790009 22042 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:47.790079 22042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:47.793888 22042 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 06:48:47.796205 22059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.797490 22042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:48:47.797614 22042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "6324d8355f0b4cde941ec554834eef50"
format_stamp: "Formatted at 2026-06-19 06:48:46 on dist-test-slave-9x3b"
server_key: "3f4d3f32b3e6eb3440258683f7bfce3c"
server_key_iv: "52908610651674476894d52dea25d1a3"
server_key_version: "encryptionkey@0"
I20260619 06:48:47.798233 22042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:47.798790 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.795010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.0.0.1:51230)
0619 06:48:47.795222 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.795238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.795397 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.795773 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.795776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.795795 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:47.795882 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.795886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.796683 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.796686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.797540 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.797550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.798383 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.798386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.798505 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.798532 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.798606 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.798669 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260619 06:48:47.798830 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.795087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.14.116.252:42545)
0619 06:48:47.795222 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.795228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.795347 (+ 119us) server_negotiation.cc:407] Connection header received
0619 06:48:47.795568 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.795572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.795656 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.795761 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:47.795993 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.796578 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.797666 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.798271 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.798395 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.798525 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.798622 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.798694 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:48:47.799386 21732 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 06:48:47.799762 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21688a106c343d58a6797563ec5ff64. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.799899 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:48:47.855949 22042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:47.859112 22042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:47.859264 22042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:47.859864 22042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:47.860647 22042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:47.860692 22042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.860731 22042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:47.860752 22042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:47.869715 22042 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:40643
I20260619 06:48:47.869830 22172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:40643 every 8 connection(s)
I20260619 06:48:47.870675 22042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:47.880079 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 22042
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.253@KRBTEST.COM
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.252@KRBTEST.COM
Jun 19 06:48:47 dist-test-slave-9x3b krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851727, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:47.887401 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.873895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:58849 (local address 127.14.116.253:34027)
0619 06:48:47.874011 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.874014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.874118 (+ 104us) server_negotiation.cc:407] Connection header received
0619 06:48:47.874568 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.874572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.874623 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.874676 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.875970 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.876493 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.878098 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.878436 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.878525 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.882358 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.882363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.882366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.882399 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.884687 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.885323 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.885328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.885332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.885424 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.885827 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.885830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.885832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.886059 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.886151 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.887085 (+ 934us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.887214 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":49}
I20260619 06:48:47.888948 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.873765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.14.116.195:58849)
0619 06:48:47.874033 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.874046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.874360 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.874841 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.874849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.875265 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.875817 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.875826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.876627 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.876629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.877953 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.877959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.878648 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.878650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.878740 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.879365 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.879390 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.882143 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.884842 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.884849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.884864 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.885201 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.885533 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.885536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.885538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.885712 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.886203 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.886211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.886519 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.886727 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.887883 (+ 1156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":275,"spinlock_wait_cycles":30208,"thread_start_us":104,"threads_started":1}
I20260619 06:48:47.889205 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.872806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.14.116.195:57307)
0619 06:48:47.873246 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.873293 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.874357 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.874840 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.874847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.875235 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.875807 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.875822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.876920 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.876923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.877894 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.877902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.878500 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.878505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.878744 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.879710 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.879725 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.882160 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.886824 (+ 4664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.886827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.886829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.887058 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.887377 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.887380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.887382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.887476 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.888338 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.888341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.888422 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.888514 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.888637 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":283,"spinlock_wait_cycles":19712,"thread_start_us":141,"threads_started":1}
I20260619 06:48:47.890421 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.872806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.195:43581)
0619 06:48:47.873244 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.873293 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.874360 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.874841 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.874849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.875261 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:47.875820 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.875826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.876878 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.876881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.877937 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.877947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.878528 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.878531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.878762 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.879597 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:47.879653 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:47.883491 (+ 3838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:47.885721 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.885724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.885726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.885954 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.887810 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:47.887815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:47.887820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:47.887901 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:47.889927 (+ 2026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:47.889932 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:47.890020 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.890120 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.890263 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":197,"spinlock_wait_cycles":20864,"thread_start_us":113,"threads_started":1}
I20260619 06:48:47.890780 22175 heartbeater.cc:347] Connected to a master server at 127.14.116.253:34027
I20260619 06:48:47.890869 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.872387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:57307 (local address 127.14.116.252:42545)
0619 06:48:47.872508 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.872512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.873391 (+ 879us) server_negotiation.cc:407] Connection header received
0619 06:48:47.874551 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.874555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.874614 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.874688 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.875947 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.876766 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.878009 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.878349 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.878450 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.882345 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.882349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.882352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.882380 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.886721 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.887165 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.887170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.887175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.887276 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.888008 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.888011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.888013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.888197 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.888298 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.890590 (+ 2292us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.890711 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:48:47.890997 22175 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:47.891327 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.872439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:43581 (local address 127.14.116.254:40519)
0619 06:48:47.872590 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.872596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.873388 (+ 792us) server_negotiation.cc:407] Connection header received
0619 06:48:47.874570 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.874573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.874623 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.874676 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.875993 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.876771 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.878080 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.878314 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.878372 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.883708 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.883713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.883718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.883750 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.885596 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.887596 (+ 2000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.887601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.887603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.887704 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.889547 (+ 1843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.889551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.889553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.889768 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.889877 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.891048 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.891166 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:48:47.891391 22173 heartbeater.cc:347] Connected to a master server at 127.14.116.252:42545
I20260619 06:48:47.891479 22173 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:47.891531 22175 heartbeater.cc:511] Master 127.14.116.253:34027 requested a full tablet report, sending...
I20260619 06:48:47.891665 22173 heartbeater.cc:511] Master 127.14.116.252:42545 requested a full tablet report, sending...
I20260619 06:48:47.892689 21719 ts_manager.cc:194] Registered new tserver with Master: 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643)
I20260619 06:48:47.892731 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.882090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.14.116.252:42545)
0619 06:48:47.882800 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.882814 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.883312 (+ 498us) server_negotiation.cc:407] Connection header received
0619 06:48:47.883352 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.883355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.883406 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.883497 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.884488 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.885209 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.886352 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.886772 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.886874 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.888721 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.888725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.888727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.888750 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.890558 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.891047 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.891050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.891051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.891131 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.891446 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.891449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.891451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.891668 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.891791 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.892404 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.892532 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":81,"threads_started":1}
I20260619 06:48:47.892935 22174 heartbeater.cc:347] Connected to a master server at 127.14.116.254:40519
I20260619 06:48:47.893008 22174 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:47.893159 22174 heartbeater.cc:511] Master 127.14.116.254:40519 requested a full tablet report, sending...
I20260619 06:48:47.893208 21529 ts_manager.cc:194] Registered new tserver with Master: 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643)
I20260619 06:48:47.894433 20935 ts_manager.cc:194] Re-registered known tserver with Master: 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643)
I20260619 06:48:47.895198 20935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:43581
I20260619 06:48:47.899075 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.884958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.14.116.253:34027)
0619 06:48:47.885415 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.885420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.885909 (+ 489us) server_negotiation.cc:407] Connection header received
0619 06:48:47.886052 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.886055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.886096 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.886151 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.887020 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.887787 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.889067 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.889472 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.889568 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.894542 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.894545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.894548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.894570 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.897502 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.897968 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.897970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.897971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.898036 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.898310 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.898315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.898317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.898479 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.898607 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.898821 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.898946 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":68,"threads_started":1}
I20260619 06:48:47.899904 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.884327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.14.116.254:40519)
0619 06:48:47.891487 (+ 7160us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.891491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.891505 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:47.891538 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.891544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.891586 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.891681 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:47.893328 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.893899 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.894829 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.895154 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.895273 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.896486 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:47.896490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:47.896493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:47.896515 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:47.898001 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.898412 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.898416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.898418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.898498 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:47.898762 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:47.898766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:47.898767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:47.898955 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:47.899081 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.899670 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.899772 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7104,"thread_start_us":73,"threads_started":1}
I20260619 06:48:47.902262 20935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:47.905288 20935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:47.907596 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: NON_VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: true } } }
I20260619 06:48:47.907753 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:47.910524 22193 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: attempting to promote NON_VOTER 2b5a2dd1ea1040b09762a53247d70e05 to VOTER
I20260619 06:48:47.910866 22193 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } }
I20260619 06:48:47.911989 21732 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 LEARNER]: Refusing update from remote peer a21688a106c343d58a6797563ec5ff64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 06:48:47.912127 21547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Refusing update from remote peer a21688a106c343d58a6797563ec5ff64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 06:48:47.912335 22191 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:48:47.912519 22193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:48:47.914216 22192 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2b5a2dd1ea1040b09762a53247d70e05 (127.14.116.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:47.914783 21547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2b5a2dd1ea1040b09762a53247d70e05 (127.14.116.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:47.914672 21732 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2b5a2dd1ea1040b09762a53247d70e05 (127.14.116.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:47.915727 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:47.915838 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:47.915974 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:47.916059 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:47.916206 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:47.916311 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:47.916419 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:47.916517 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:48:47.917038 22201 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21688a106c343d58a6797563ec5ff64" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:47.917148 22201 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:48:47.927428 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.921675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:40389 (local address 127.0.0.1:44332)
0619 06:48:47.922147 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.922321 (+ 174us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.922412 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.923283 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.923286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.923302 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:47.923410 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.923416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.924753 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.924757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.925856 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.925868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.926939 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.926943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.927105 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.927134 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.927219 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.927294 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":65}
I20260619 06:48:47.927430 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.921921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:40643 (local address 127.0.0.1:44428)
0619 06:48:47.922153 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.922171 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.922249 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.922960 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.922963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.922979 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:47.923070 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.923076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.924692 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.924696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.925952 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.925963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.926923 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.926927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.927052 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.927122 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.927202 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.927285 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":75,"thread_start_us":88,"threads_started":1}
I20260619 06:48:47.927474 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.922084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.14.116.195:40643)
0619 06:48:47.922470 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.922483 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.922509 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:48:47.922589 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.922594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.922802 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.923225 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:47.923367 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.924546 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.926077 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.926808 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.926941 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.927076 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.927253 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.927345 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":87,"threads_started":1}
I20260619 06:48:47.927845 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.922215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:45885 (local address 127.0.0.1:41396)
0619 06:48:47.922999 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.923011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.923097 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.923670 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.923673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.923694 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:47.923774 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.923780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.925057 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.925060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.926394 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.926402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.927418 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.927421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.927515 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.927537 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.927614 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.927718 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":383,"thread_start_us":78,"threads_started":1}
I20260619 06:48:47.928261 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.922416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.14.116.194:45885)
0619 06:48:47.922735 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.922747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.923186 (+ 439us) server_negotiation.cc:407] Connection header received
0619 06:48:47.923290 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.923295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.923517 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.923689 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:47.923893 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.924935 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.926500 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.927143 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.927251 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.927361 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.928045 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.928135 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":112,"threads_started":1}
I20260619 06:48:47.928735 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.921817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.14.116.193:40389)
0619 06:48:47.922803 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.922818 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.922838 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:48:47.922925 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.922933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.923123 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.923267 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:47.923500 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.924633 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.925999 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.926801 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.928329 (+ 1528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.928453 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.928502 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.928591 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":109,"threads_started":1}
I20260619 06:48:47.930474 22107 tablet_service.cc:1467] Processing CreateTablet for tablet 32db81cb73434034a3d27fb70f0b616e (DEFAULT_TABLE table=test-table [id=ae1158f7eeca4b5886d81ad8c67d5c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:47.930660 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 32db81cb73434034a3d27fb70f0b616e (DEFAULT_TABLE table=test-table [id=ae1158f7eeca4b5886d81ad8c67d5c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:47.930799 21832 tablet_service.cc:1467] Processing CreateTablet for tablet 32db81cb73434034a3d27fb70f0b616e (DEFAULT_TABLE table=test-table [id=ae1158f7eeca4b5886d81ad8c67d5c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:47.931576 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32db81cb73434034a3d27fb70f0b616e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:47.931576 22107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32db81cb73434034a3d27fb70f0b616e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:47.931712 21832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32db81cb73434034a3d27fb70f0b616e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:47.938170 22207 tablet_bootstrap.cc:492] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: Bootstrap starting.
I20260619 06:48:47.939486 22209 tablet_bootstrap.cc:492] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: Bootstrap starting.
I20260619 06:48:47.940016 22208 tablet_bootstrap.cc:492] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: Bootstrap starting.
I20260619 06:48:47.940729 22207 tablet_bootstrap.cc:654] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:47.941463 22207 log.cc:824] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:47.942752 22208 tablet_bootstrap.cc:654] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:47.942921 22209 tablet_bootstrap.cc:654] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:47.943477 22208 log.cc:824] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:47.943702 22209 log.cc:824] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:47.943769 22207 tablet_bootstrap.cc:492] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: No bootstrap required, opened a new log
I20260619 06:48:47.944029 22207 ts_tablet_manager.cc:1397] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:48:47.945330 22208 tablet_bootstrap.cc:492] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: No bootstrap required, opened a new log
I20260619 06:48:47.945488 22208 ts_tablet_manager.cc:1397] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:48:47.945554 22209 tablet_bootstrap.cc:492] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: No bootstrap required, opened a new log
I20260619 06:48:47.945695 22209 ts_tablet_manager.cc:1397] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 06:48:47.947965 22207 raft_consensus.cc:348] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } }
I20260619 06:48:47.948206 22207 raft_consensus.cc:374] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:47.948253 22207 raft_consensus.cc:729] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890b226d9d0b460fbc14beb457ba427a, State: Initialized, Role: FOLLOWER
I20260619 06:48:47.948798 22207 consensus_queue.cc:260] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } }
I20260619 06:48:47.949756 22207 ts_tablet_manager.cc:1428] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:48:47.949687 22208 raft_consensus.cc:348] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } }
I20260619 06:48:47.949913 22208 raft_consensus.cc:374] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:47.949957 22208 raft_consensus.cc:729] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6324d8355f0b4cde941ec554834eef50, State: Initialized, Role: FOLLOWER
I20260619 06:48:47.949918 22209 raft_consensus.cc:348] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } }
I20260619 06:48:47.950129 22209 raft_consensus.cc:374] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:47.950172 22209 raft_consensus.cc:729] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc0643f08674c3e8fa38ab673b06a96, State: Initialized, Role: FOLLOWER
I20260619 06:48:47.950361 22208 consensus_queue.cc:260] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } }
I20260619 06:48:47.950564 22209 consensus_queue.cc:260] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } }
I20260619 06:48:47.951207 22208 ts_tablet_manager.cc:1428] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:48:47.951352 22209 ts_tablet_manager.cc:1428] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:48:47.951828 22036 heartbeater.cc:503] Master 127.14.116.254:40519 was elected leader, sending a full tablet report...
I20260619 06:48:47.952536 22174 heartbeater.cc:503] Master 127.14.116.254:40519 was elected leader, sending a full tablet report...
I20260619 06:48:47.954885 21899 heartbeater.cc:503] Master 127.14.116.254:40519 was elected leader, sending a full tablet report...
I20260619 06:48:47.978754 22215 raft_consensus.cc:482] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:47.978964 22215 raft_consensus.cc:504] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } }
I20260619 06:48:47.980096 22215 leader_election.cc:288] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 890b226d9d0b460fbc14beb457ba427a (127.14.116.193:40389), 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643)
I20260619 06:48:47.984880 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.980681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:52243 (local address 127.14.116.195:40643)
0619 06:48:47.980817 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.980821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.980835 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:47.980882 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.980888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.980933 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.980990 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:47.981235 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.981948 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.983599 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.984258 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.984372 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.984425 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.984704 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.984765 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:48:47.984880 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.980412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:50481 (local address 127.14.116.193:40389)
0619 06:48:47.980554 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:47.980558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:47.980575 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:48:47.980637 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:47.980640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:47.980691 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:47.980795 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:47.981102 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.981904 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.983599 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.984258 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.984372 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.984423 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:47.984682 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 06:48:47.984754 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:48:47.984896 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.980304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:40389 (local address 127.14.116.194:50481)
0619 06:48:47.980431 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.980449 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.980557 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.980801 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.980804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.980821 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:47.980983 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.980992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.982021 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.982025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.983481 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.983492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.984389 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.984393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.984525 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.984567 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.984652 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.984736 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":102}
I20260619 06:48:47.984896 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:47.980593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:40643 (local address 127.14.116.194:52243)
0619 06:48:47.980709 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:47.980723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:47.980794 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:47.981021 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:47.981026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:47.981048 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:47.981146 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.981150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.982032 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.982034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.983481 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:47.983492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:47.984389 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:47.984393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:47.984574 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:47.984591 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 06:48:47.984690 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:48:47.984742 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":121}
I20260619 06:48:47.985773 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32db81cb73434034a3d27fb70f0b616e" candidate_uuid: "ecc0643f08674c3e8fa38ab673b06a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890b226d9d0b460fbc14beb457ba427a" is_pre_election: true
I20260619 06:48:47.985875 22127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32db81cb73434034a3d27fb70f0b616e" candidate_uuid: "ecc0643f08674c3e8fa38ab673b06a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6324d8355f0b4cde941ec554834eef50" is_pre_election: true
I20260619 06:48:47.986104 21852 raft_consensus.cc:2454] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc0643f08674c3e8fa38ab673b06a96 in term 0.
I20260619 06:48:47.986135 22127 raft_consensus.cc:2454] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc0643f08674c3e8fa38ab673b06a96 in term 0.
I20260619 06:48:47.986555 21925 leader_election.cc:302] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890b226d9d0b460fbc14beb457ba427a, ecc0643f08674c3e8fa38ab673b06a96; no voters:
I20260619 06:48:47.986811 22215 raft_consensus.cc:2781] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:47.986919 22215 raft_consensus.cc:482] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:47.986970 22215 raft_consensus.cc:3037] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:47.988350 22215 raft_consensus.cc:504] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } }
I20260619 06:48:47.988770 22215 leader_election.cc:288] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [CANDIDATE]: Term 1 election: Requested vote from peers 890b226d9d0b460fbc14beb457ba427a (127.14.116.193:40389), 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643)
I20260619 06:48:47.989070 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32db81cb73434034a3d27fb70f0b616e" candidate_uuid: "ecc0643f08674c3e8fa38ab673b06a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890b226d9d0b460fbc14beb457ba427a"
I20260619 06:48:47.989140 22127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32db81cb73434034a3d27fb70f0b616e" candidate_uuid: "ecc0643f08674c3e8fa38ab673b06a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6324d8355f0b4cde941ec554834eef50"
I20260619 06:48:47.989202 21852 raft_consensus.cc:3037] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:47.989253 22127 raft_consensus.cc:3037] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:47.990478 22127 raft_consensus.cc:2454] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc0643f08674c3e8fa38ab673b06a96 in term 1.
I20260619 06:48:47.990612 21852 raft_consensus.cc:2454] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc0643f08674c3e8fa38ab673b06a96 in term 1.
I20260619 06:48:47.990741 21925 leader_election.cc:302] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6324d8355f0b4cde941ec554834eef50, ecc0643f08674c3e8fa38ab673b06a96; no voters:
I20260619 06:48:47.990945 22215 raft_consensus.cc:2781] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:47.991207 22215 raft_consensus.cc:686] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 1 LEADER]: Becoming Leader. State: Replica: ecc0643f08674c3e8fa38ab673b06a96, State: Running, Role: LEADER
I20260619 06:48:47.991588 22215 consensus_queue.cc:237] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } }
I20260619 06:48:47.995090 20934 catalog_manager.cc:5495] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc0643f08674c3e8fa38ab673b06a96 (127.14.116.194). New cstate: current_term: 1 leader_uuid: "ecc0643f08674c3e8fa38ab673b06a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } health_report { overall_health: HEALTHY } } }
I20260619 06:48:48.011754 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:48.008234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.14.116.194:45885)
0619 06:48:48.008364 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:48.008370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:48.008383 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:48:48.008573 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:48.008575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:48.008627 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:48.008728 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:48.008970 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.009477 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.010379 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.010597 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.010698 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:48.010981 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:48.011340 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:48.011472 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:48.011535 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 06:48:48.011588 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:48:48.017555 21852 raft_consensus.cc:1261] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [term 1 FOLLOWER]: Refusing update from remote peer ecc0643f08674c3e8fa38ab673b06a96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:48.017560 22127 raft_consensus.cc:1261] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [term 1 FOLLOWER]: Refusing update from remote peer ecc0643f08674c3e8fa38ab673b06a96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:48.018232 22216 consensus_queue.cc:1035] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:48.018395 22215 consensus_queue.cc:1035] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:48.029191 22220 mvcc.cc:205] Tried to move back new op lower bound from 7298464677953384448 to 7298464677856747520. Current Snapshot: MvccSnapshot[applied={T|T < 7298464677953384448}]
I20260619 06:48:48.032202 22221 mvcc.cc:205] Tried to move back new op lower bound from 7298464677953384448 to 7298464677856747520. Current Snapshot: MvccSnapshot[applied={T|T < 7298464677953384448}]
I20260619 06:48:48.036599 20934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:48.036725 20934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:48.041263 20934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a21688a106c343d58a6797563ec5ff64: Sending DeleteTablet for 3 replicas of tablet 32db81cb73434034a3d27fb70f0b616e
I20260619 06:48:48.041949 21832 tablet_service.cc:1514] Processing DeleteTablet for tablet 32db81cb73434034a3d27fb70f0b616e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:48 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:44332
I20260619 06:48:48.042158 22107 tablet_service.cc:1514] Processing DeleteTablet for tablet 32db81cb73434034a3d27fb70f0b616e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:48 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:44428
I20260619 06:48:48.042416 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 32db81cb73434034a3d27fb70f0b616e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:48 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:41396
I20260619 06:48:48.042850 22230 tablet_replica.cc:330] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: stopping tablet replica
I20260619 06:48:48.042915 22228 tablet_replica.cc:330] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: stopping tablet replica
I20260619 06:48:48.043351 22230 raft_consensus.cc:2229] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:48.043386 22228 raft_consensus.cc:2229] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:48.043717 22228 raft_consensus.cc:2258] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:48.043924 22230 raft_consensus.cc:2258] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:48.044492 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 20903
I20260619 06:48:48.045807 22228 ts_tablet_manager.cc:1905] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:48.046254 22230 ts_tablet_manager.cc:1905] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:48.048825 22230 ts_tablet_manager.cc:1918] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:48.048980 22230 log.cc:1196] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/32db81cb73434034a3d27fb70f0b616e
I20260619 06:48:48.049173 22228 ts_tablet_manager.cc:1918] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:48.049275 22228 log.cc:1196] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/32db81cb73434034a3d27fb70f0b616e
I20260619 06:48:48.049458 22230 ts_tablet_manager.cc:1939] T 32db81cb73434034a3d27fb70f0b616e P ecc0643f08674c3e8fa38ab673b06a96: Deleting consensus metadata
I20260619 06:48:48.049582 22228 ts_tablet_manager.cc:1939] T 32db81cb73434034a3d27fb70f0b616e P 890b226d9d0b460fbc14beb457ba427a: Deleting consensus metadata
W20260619 06:48:48.051527 21925 connection.cc:535] server connection from 127.0.0.1:41396 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 06:48:48.051723 21788 connection.cc:535] server connection from 127.0.0.1:44332 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:48:48.052688 22229 tablet_replica.cc:330] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: stopping tablet replica
I20260619 06:48:48.053031 22229 raft_consensus.cc:2229] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:48.053351 22229 raft_consensus.cc:2258] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:48.054790 22229 ts_tablet_manager.cc:1905] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:48.057467 22229 ts_tablet_manager.cc:1918] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:48.057574 22229 log.cc:1196] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/32db81cb73434034a3d27fb70f0b616e
I20260619 06:48:48.057945 22229 ts_tablet_manager.cc:1939] T 32db81cb73434034a3d27fb70f0b616e P 6324d8355f0b4cde941ec554834eef50: Deleting consensus metadata
W20260619 06:48:48.059106 22062 connection.cc:441] server connection from 127.0.0.1:44428 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44428 (request call id 1) could send its response
I20260619 06:48:48.062077 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:48.043385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.14.116.253:34027)
0619 06:48:48.043487 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:48.043491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:48.044268 (+ 777us) server_negotiation.cc:407] Connection header received
0619 06:48:48.050919 (+ 6651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:48.050925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:48.050988 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:48.051045 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:48.054281 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.055041 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.056244 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.056579 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.056656 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:48.058667 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:48.058671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:48.058673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:48.058699 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:48.060386 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:48.060798 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:48.060802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:48.060804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:48.060896 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:48.061171 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:48.061174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:48.061176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:48.061382 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:48.061511 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:48.061780 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 06:48:48.061906 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:48:48.064632 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:48.054526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.14.116.253:34027)
0619 06:48:48.054626 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:48.054631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:48.055366 (+ 735us) server_negotiation.cc:407] Connection header received
0619 06:48:48.055522 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:48.055525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:48.055567 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:48.055632 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:48.056549 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.057316 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.058632 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.058985 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.059055 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:48.061122 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:48.061127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:48.061131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:48.061159 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:48.062801 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:48.063296 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:48.063300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:48.063313 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:48.063390 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:48.063737 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:48.063741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:48.063742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:48.063930 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:48.064042 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:48.064382 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 06:48:48.064491 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:48:48.070250 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:48.043165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51246 (local address 127.14.116.252:42545)
0619 06:48:48.043273 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:48.043277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:48.062157 (+ 18880us) server_negotiation.cc:407] Connection header received
0619 06:48:48.062355 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:48.062361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:48.062421 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:48.062475 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:48.063301 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.064046 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.065119 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.065434 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.065514 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:48.066878 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:48.066882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:48.066885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:48.066909 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:48.068542 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:48.068956 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:48.068959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:48.068961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:48.069042 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:48.069290 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:48.069296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:48.069299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:48.069538 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:48.069667 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:48.069896 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 06:48:48.070040 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":29}
I20260619 06:48:48.072105 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:48.063933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.14.116.252:42545)
0619 06:48:48.064044 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:48.064051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:48.064083 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:48:48.064195 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:48.064198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:48.064238 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:48.064301 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:48.065239 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.065997 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.067086 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:48.067405 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:48.067516 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:48.069054 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:48.069059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:48.069065 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:48.069092 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:48.070626 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:48.071065 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:48.071069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:48.071071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:48.071149 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:48.071410 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:48.071413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:48.071414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:48.071577 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:48.071742 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:48.071932 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 06:48:48.072010 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:48:49.032194 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.031733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.195:48065)
0619 06:48:49.031975 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.032084 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
I20260619 06:48:49.032733 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.032107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:52039)
0619 06:48:49.032539 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.032624 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":63,"threads_started":1}
W20260619 06:48:49.032823 22174 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:40519 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
W20260619 06:48:49.032974 22036 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:40519 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:49.033295 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.032749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.193:35081)
0619 06:48:49.033152 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.033224 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":55,"threads_started":1}
W20260619 06:48:49.033515 21899 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:40519 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:49.540777 22241 raft_consensus.cc:482] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a21688a106c343d58a6797563ec5ff64)
I20260619 06:48:49.541013 22241 raft_consensus.cc:504] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } }
I20260619 06:48:49.541147 22242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a21688a106c343d58a6797563ec5ff64)
I20260619 06:48:49.541363 22242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } }
I20260619 06:48:49.542124 22241 leader_election.cc:288] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519), 2b5a2dd1ea1040b09762a53247d70e05 (127.14.116.252:42545)
I20260619 06:48:49.542325 22242 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519), afb9debb2e264c13b53af4b570720a01 (127.14.116.253:34027)
I20260619 06:48:49.542801 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.542280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:38386)
0619 06:48:49.542601 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.542688 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":56,"threads_started":1}
W20260619 06:48:49.542987 21523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:49.543077 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.542616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:38388)
0619 06:48:49.542907 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.542974 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":53,"threads_started":1}
W20260619 06:48:49.543224 21705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:49.543913 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.543654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:38396)
0619 06:48:49.543781 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.543822 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 06:48:49.544044 21523 leader_election.cc:334] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519): Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:49.544128 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.543895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:38410)
0619 06:48:49.544022 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.544067 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 06:48:49.544221 21705 leader_election.cc:334] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519): Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:49.546716 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.542321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:42545 (local address 127.0.0.1:51268)
0619 06:48:49.542618 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.542633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:49.542732 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:49.543272 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:49.543275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:49.543296 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:49.543418 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:49.543430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:49.544748 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:49.544751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:49.545612 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:49.545620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:49.546317 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:49.546320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:49.546404 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:49.546442 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:48:49.546515 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 06:48:49.546580 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260619 06:48:49.546723 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.542492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.14.116.252:42545)
0619 06:48:49.542776 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:49.542783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:49.542798 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:48:49.542948 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:49.542954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:49.543004 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:49.543573 (+ 569us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:49.543699 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:49.544632 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:49.545722 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:49.546193 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:49.546275 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:49.546314 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:49.546536 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:48:49.546601 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":52,"threads_started":1}
I20260619 06:48:49.546952 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.543140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.14.116.253:34027)
0619 06:48:49.543721 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:49.543730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:49.543755 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:48:49.543793 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:49.543795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:49.543835 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:49.543930 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:49.544284 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:49.544932 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:49.545952 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:49.546434 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:49.546542 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:49.546579 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:49.546753 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 06:48:49.546826 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":98,"threads_started":1}
I20260619 06:48:49.547124 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.542887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:34027 (local address 127.0.0.1:38248)
0619 06:48:49.543389 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.543404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:49.543489 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:49.544039 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:49.544042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:49.544062 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:49.544151 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:49.544157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:49.545032 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:49.545035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:49.545830 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:49.545838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:49.546544 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:49.546547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:49.546625 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:49.546645 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:48:49.546719 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:48:49.546847 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":61,"threads_started":1}
I20260619 06:48:49.547129 21732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb9debb2e264c13b53af4b570720a01" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2b5a2dd1ea1040b09762a53247d70e05" is_pre_election: true
I20260619 06:48:49.547345 21732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb9debb2e264c13b53af4b570720a01 in term 1.
I20260619 06:48:49.547525 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b5a2dd1ea1040b09762a53247d70e05" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "afb9debb2e264c13b53af4b570720a01" is_pre_election: true
I20260619 06:48:49.547710 21522 leader_election.cc:302] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b5a2dd1ea1040b09762a53247d70e05, afb9debb2e264c13b53af4b570720a01; no voters: a21688a106c343d58a6797563ec5ff64
I20260619 06:48:49.547739 21547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b5a2dd1ea1040b09762a53247d70e05 in term 1.
I20260619 06:48:49.547935 22241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 06:48:49.547986 22241 raft_consensus.cc:482] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a21688a106c343d58a6797563ec5ff64)
I20260619 06:48:49.548025 22241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:48:49.548058 21705 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b5a2dd1ea1040b09762a53247d70e05, afb9debb2e264c13b53af4b570720a01; no voters: a21688a106c343d58a6797563ec5ff64
I20260619 06:48:49.548250 22242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 06:48:49.548305 22242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a21688a106c343d58a6797563ec5ff64)
I20260619 06:48:49.548354 22242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:48:49.549073 22241 raft_consensus.cc:504] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } }
I20260619 06:48:49.549286 22242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } }
I20260619 06:48:49.549430 22241 leader_election.cc:288] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [CANDIDATE]: Term 2 election: Requested vote from peers a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519), 2b5a2dd1ea1040b09762a53247d70e05 (127.14.116.252:42545)
I20260619 06:48:49.549639 22242 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 2 election: Requested vote from peers a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519), afb9debb2e264c13b53af4b570720a01 (127.14.116.253:34027)
I20260619 06:48:49.549701 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.549502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:38412)
0619 06:48:49.549592 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.549642 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 06:48:49.549861 21732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb9debb2e264c13b53af4b570720a01" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2b5a2dd1ea1040b09762a53247d70e05"
I20260619 06:48:49.549961 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.549728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:38426)
0619 06:48:49.549819 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.549868 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 06:48:49.550024 21732 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate afb9debb2e264c13b53af4b570720a01 in current term 2: Already voted for candidate 2b5a2dd1ea1040b09762a53247d70e05 in this term.
I20260619 06:48:49.550046 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b5a2dd1ea1040b09762a53247d70e05" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "afb9debb2e264c13b53af4b570720a01"
I20260619 06:48:49.550189 21547 raft_consensus.cc:2379] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b5a2dd1ea1040b09762a53247d70e05 in current term 2: Already voted for candidate afb9debb2e264c13b53af4b570720a01 in this term.
I20260619 06:48:49.550673 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.550476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:38432)
0619 06:48:49.550572 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.550620 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:48:49.550758 21523 leader_election.cc:334] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519): Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:49.550827 21523 leader_election.cc:302] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afb9debb2e264c13b53af4b570720a01; no voters: 2b5a2dd1ea1040b09762a53247d70e05, a21688a106c343d58a6797563ec5ff64
I20260619 06:48:49.550952 22241 raft_consensus.cc:2726] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 06:48:49.551009 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:49.550802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:38436)
0619 06:48:49.550896 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:49.550946 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:48:49.551092 21705 leader_election.cc:334] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519): Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:49.551174 21705 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b5a2dd1ea1040b09762a53247d70e05; no voters: a21688a106c343d58a6797563ec5ff64, afb9debb2e264c13b53af4b570720a01
I20260619 06:48:49.551283 22242 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 06:48:50.033849 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:50.033401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:53143)
0619 06:48:50.033658 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:50.033727 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":52,"threads_started":1}
I20260619 06:48:50.033914 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:50.033359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.195:37079)
0619 06:48:50.033775 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:50.033826 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":75,"threads_started":1}
I20260619 06:48:50.034462 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:50.034040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.193:45441)
0619 06:48:50.034338 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:50.034392 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":65,"threads_started":1}
I20260619 06:48:51.034907 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.034457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.195:34195)
0619 06:48:51.034723 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.034783 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":71,"threads_started":1}
I20260619 06:48:51.035218 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.034458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:40977)
0619 06:48:51.035060 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.035125 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":74,"threads_started":1}
I20260619 06:48:51.035640 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.035038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.193:35099)
0619 06:48:51.035462 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.035526 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":60,"threads_started":1}
I20260619 06:48:51.629859 22257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:51.629997 22257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } }
I20260619 06:48:51.630420 22257 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519), afb9debb2e264c13b53af4b570720a01 (127.14.116.253:34027)
I20260619 06:48:51.631000 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.630633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:59336)
0619 06:48:51.630832 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.630908 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260619 06:48:51.631196 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b5a2dd1ea1040b09762a53247d70e05" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "afb9debb2e264c13b53af4b570720a01" is_pre_election: true
I20260619 06:48:51.631354 21547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b5a2dd1ea1040b09762a53247d70e05 in term 2.
I20260619 06:48:51.631742 21705 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b5a2dd1ea1040b09762a53247d70e05, afb9debb2e264c13b53af4b570720a01; no voters:
I20260619 06:48:51.631826 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.631582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:59348)
0619 06:48:51.631699 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.631751 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:48:51.631915 22257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260619 06:48:51.631961 21705 leader_election.cc:334] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519): Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:51.631976 22257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:51.632051 22257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:48:51.633096 22257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } }
I20260619 06:48:51.633436 22257 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 3 election: Requested vote from peers a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519), afb9debb2e264c13b53af4b570720a01 (127.14.116.253:34027)
I20260619 06:48:51.633754 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.633526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:59358)
0619 06:48:51.633618 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.633676 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 06:48:51.633889 21547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b5a2dd1ea1040b09762a53247d70e05" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "afb9debb2e264c13b53af4b570720a01"
I20260619 06:48:51.634008 21547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:48:51.634367 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.634177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:59366)
0619 06:48:51.634275 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.634318 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 06:48:51.634465 21705 leader_election.cc:334] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519): Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
I20260619 06:48:51.635007 21547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b5a2dd1ea1040b09762a53247d70e05 in term 3.
I20260619 06:48:51.635286 21705 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b5a2dd1ea1040b09762a53247d70e05, afb9debb2e264c13b53af4b570720a01; no voters: a21688a106c343d58a6797563ec5ff64
I20260619 06:48:51.635473 22257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:48:51.635685 22257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [term 3 LEADER]: Becoming Leader. State: Replica: 2b5a2dd1ea1040b09762a53247d70e05, State: Running, Role: LEADER
I20260619 06:48:51.635939 22257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } }
I20260619 06:48:51.637238 22260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5a2dd1ea1040b09762a53247d70e05. Latest consensus state: current_term: 3 leader_uuid: "2b5a2dd1ea1040b09762a53247d70e05" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:51.637352 22260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:51.637599 22262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:51.639770 22262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:51.640143 22262 catalog_manager.cc:1259] Loaded cluster ID: 22e17090ec044d4086624ef9fca979b3
I20260619 06:48:51.640206 22262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:51.640755 22262 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:51.641121 22262 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Loaded TSK: 0
I20260619 06:48:51.641812 22262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:48:51.685097 21718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:51.688149 21718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:51.691030 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.690791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:59384)
0619 06:48:51.690916 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.690968 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:48:51.691020 21547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [term 3 FOLLOWER]: Refusing update from remote peer 2b5a2dd1ea1040b09762a53247d70e05: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 06:48:51.691480 22260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 06:48:51.691766 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.691538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:59396)
0619 06:48:51.691671 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.691715 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 06:48:51.691872 21705 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 -> Peer a21688a106c343d58a6797563ec5ff64 (127.14.116.254:40519): Couldn't send request to peer a21688a106c343d58a6797563ec5ff64. Status: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:48:51.692878 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5a2dd1ea1040b09762a53247d70e05. Latest consensus state: current_term: 3 leader_uuid: "2b5a2dd1ea1040b09762a53247d70e05" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:51.693002 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:48:51.694131 22260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2b5a2dd1ea1040b09762a53247d70e05" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:51.694243 22260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:51.694325 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2b5a2dd1ea1040b09762a53247d70e05" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21688a106c343d58a6797563ec5ff64" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40519 } } peers { permanent_uuid: "afb9debb2e264c13b53af4b570720a01" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 34027 } attrs { promote: false } } peers { permanent_uuid: "2b5a2dd1ea1040b09762a53247d70e05" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 42545 } attrs { promote: false } } }
I20260619 06:48:51.694453 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9debb2e264c13b53af4b570720a01 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:48:51.704716 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.699717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:40643 (local address 127.0.0.1:60376)
0619 06:48:51.699862 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.699877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:51.699983 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:51.701149 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:51.701152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:51.701166 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:51.701254 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:51.701260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.702037 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.702043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:51.703650 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:51.703657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.704270 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.704273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:51.704406 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:51.704429 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:48:51.704502 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 06:48:51.704578 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":283}
I20260619 06:48:51.704900 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.699959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:45885 (local address 127.0.0.1:32814)
0619 06:48:51.700185 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.700197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:51.700278 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:51.701398 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:51.701401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:51.701417 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:51.701515 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:51.701520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.702497 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.702503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:51.703535 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:51.703547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.704406 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.704409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:51.704520 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:51.704547 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:48:51.704616 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 06:48:51.704685 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260619 06:48:51.704954 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.700361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.14.116.193:40389)
0619 06:48:51.700579 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:51.700583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:51.700597 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:51.700676 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:51.700679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:51.700722 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:51.700821 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:51.701082 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.701888 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.703569 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.704201 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.704414 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:51.704475 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:51.704633 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 06:48:51.704707 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
I20260619 06:48:51.705209 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.700680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.14.116.194:45885)
0619 06:48:51.701172 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:51.701177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:51.701188 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:48:51.701233 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:51.701236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:51.701280 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:51.701615 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:51.701721 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.702389 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.703686 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.704299 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.704803 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:51.704905 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:51.704949 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 06:48:51.705074 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":73,"threads_started":1}
I20260619 06:48:51.705214 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.700262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:40389 (local address 127.0.0.1:60456)
0619 06:48:51.700501 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.700514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:51.700591 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:51.700827 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:51.700830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:51.700846 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:51.700937 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:51.700942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.702015 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.702018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:51.703413 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:51.703426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.704335 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.704341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:51.704488 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:51.704542 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 06:48:51.704605 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 06:48:51.704659 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":146,"thread_start_us":87,"threads_started":1}
I20260619 06:48:51.705681 21969 tablet_service.cc:1467] Processing CreateTablet for tablet 385c826e6c784c2c87b5c3d216f56d4c (DEFAULT_TABLE table=test-table [id=bb1b5e560b544b0897723ced239481c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:51.705960 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385c826e6c784c2c87b5c3d216f56d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:51.705907 21832 tablet_service.cc:1467] Processing CreateTablet for tablet 385c826e6c784c2c87b5c3d216f56d4c (DEFAULT_TABLE table=test-table [id=bb1b5e560b544b0897723ced239481c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:51.706197 21832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385c826e6c784c2c87b5c3d216f56d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:51.708659 22275 tablet_bootstrap.cc:492] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a: Bootstrap starting.
I20260619 06:48:51.708667 22276 tablet_bootstrap.cc:492] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: Bootstrap starting.
I20260619 06:48:51.708693 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.699833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.14.116.195:40643)
0619 06:48:51.700941 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:51.700945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:51.700959 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:51.701007 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:51.701009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:51.701044 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:51.701139 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:51.701364 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.701938 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.703739 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:51.704159 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:51.704284 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:51.704338 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:51.705246 (+ 908us) server_negotiation.cc:299] Negotiation successful
0619 06:48:51.705349 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":107,"threads_started":1}
I20260619 06:48:51.709168 22107 tablet_service.cc:1467] Processing CreateTablet for tablet 385c826e6c784c2c87b5c3d216f56d4c (DEFAULT_TABLE table=test-table [id=bb1b5e560b544b0897723ced239481c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:51.709421 22107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385c826e6c784c2c87b5c3d216f56d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:51.709581 22276 tablet_bootstrap.cc:654] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:51.710071 22275 tablet_bootstrap.cc:654] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:51.710763 22276 tablet_bootstrap.cc:492] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: No bootstrap required, opened a new log
I20260619 06:48:51.710872 22276 ts_tablet_manager.cc:1397] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:51.711284 22276 raft_consensus.cc:348] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:51.711400 22276 raft_consensus.cc:374] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:51.711453 22276 raft_consensus.cc:729] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc0643f08674c3e8fa38ab673b06a96, State: Initialized, Role: FOLLOWER
I20260619 06:48:51.711575 22276 consensus_queue.cc:260] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:51.711786 22275 tablet_bootstrap.cc:492] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a: No bootstrap required, opened a new log
I20260619 06:48:51.711860 22275 ts_tablet_manager.cc:1397] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:48:51.711925 22276 ts_tablet_manager.cc:1428] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:51.712289 22275 raft_consensus.cc:348] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:51.712405 22275 raft_consensus.cc:374] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:51.712448 22275 raft_consensus.cc:729] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890b226d9d0b460fbc14beb457ba427a, State: Initialized, Role: FOLLOWER
I20260619 06:48:51.712575 22275 consensus_queue.cc:260] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:51.712972 22275 ts_tablet_manager.cc:1428] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:51.713166 22279 tablet_bootstrap.cc:492] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: Bootstrap starting.
I20260619 06:48:51.713624 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.713144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.193:34737)
0619 06:48:51.713471 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.713525 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":81,"threads_started":1}
I20260619 06:48:51.714039 22279 tablet_bootstrap.cc:654] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:51.714676 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.713991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.193:51135)
0619 06:48:51.714545 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.714600 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":74,"threads_started":1}
I20260619 06:48:51.715412 22279 tablet_bootstrap.cc:492] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: No bootstrap required, opened a new log
I20260619 06:48:51.715406 21898 heartbeater.cc:503] Master 127.14.116.252:42545 was elected leader, sending a full tablet report...
I20260619 06:48:51.715494 22279 ts_tablet_manager.cc:1397] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:51.716173 22035 heartbeater.cc:503] Master 127.14.116.252:42545 was elected leader, sending a full tablet report...
I20260619 06:48:51.716217 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.712303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:40811)
0619 06:48:51.716047 (+ 3744us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.716106 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3680,"thread_start_us":104,"threads_started":1}
I20260619 06:48:51.716593 22279 raft_consensus.cc:348] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:51.716697 22279 raft_consensus.cc:374] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:51.716740 22279 raft_consensus.cc:729] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6324d8355f0b4cde941ec554834eef50, State: Initialized, Role: FOLLOWER
I20260619 06:48:51.716881 22279 consensus_queue.cc:260] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:51.717101 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.716713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:57625)
0619 06:48:51.716978 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.717027 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
I20260619 06:48:51.717427 22279 ts_tablet_manager.cc:1428] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:51.717993 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.717520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.195:40521)
0619 06:48:51.717843 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.717896 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":83,"threads_started":1}
I20260619 06:48:51.718596 22173 heartbeater.cc:503] Master 127.14.116.252:42545 was elected leader, sending a full tablet report...
I20260619 06:48:51.718684 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:51.718411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.195:52517)
0619 06:48:51.718551 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:51.718596 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 06:48:51.874554 22176 tablet.cc:2369] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:48:51.889952 22038 tablet.cc:2369] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:48:51.949014 21901 tablet.cc:2369] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:52.077482 22282 raft_consensus.cc:482] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:52.077623 22282 raft_consensus.cc:504] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:52.078636 22282 leader_election.cc:288] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643), ecc0643f08674c3e8fa38ab673b06a96 (127.14.116.194:45885)
I20260619 06:48:52.078647 22280 raft_consensus.cc:482] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:52.078774 22280 raft_consensus.cc:504] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:52.079191 22280 leader_election.cc:288] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643), 890b226d9d0b460fbc14beb457ba427a (127.14.116.193:40389)
I20260619 06:48:52.079660 22127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385c826e6c784c2c87b5c3d216f56d4c" candidate_uuid: "ecc0643f08674c3e8fa38ab673b06a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6324d8355f0b4cde941ec554834eef50" is_pre_election: true
I20260619 06:48:52.079705 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385c826e6c784c2c87b5c3d216f56d4c" candidate_uuid: "ecc0643f08674c3e8fa38ab673b06a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890b226d9d0b460fbc14beb457ba427a" is_pre_election: true
I20260619 06:48:52.079808 22127 raft_consensus.cc:2454] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc0643f08674c3e8fa38ab673b06a96 in term 0.
I20260619 06:48:52.079823 21852 raft_consensus.cc:2454] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc0643f08674c3e8fa38ab673b06a96 in term 0.
I20260619 06:48:52.080147 21925 leader_election.cc:302] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6324d8355f0b4cde941ec554834eef50, ecc0643f08674c3e8fa38ab673b06a96; no voters:
I20260619 06:48:52.080372 22280 raft_consensus.cc:2781] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:52.080439 22280 raft_consensus.cc:482] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:52.080476 22280 raft_consensus.cc:3037] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:52.081660 22280 raft_consensus.cc:504] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:52.081995 22280 leader_election.cc:288] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [CANDIDATE]: Term 1 election: Requested vote from peers 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643), 890b226d9d0b460fbc14beb457ba427a (127.14.116.193:40389)
I20260619 06:48:52.082340 21852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385c826e6c784c2c87b5c3d216f56d4c" candidate_uuid: "ecc0643f08674c3e8fa38ab673b06a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890b226d9d0b460fbc14beb457ba427a"
I20260619 06:48:52.082443 21852 raft_consensus.cc:3037] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:52.082489 22127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385c826e6c784c2c87b5c3d216f56d4c" candidate_uuid: "ecc0643f08674c3e8fa38ab673b06a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6324d8355f0b4cde941ec554834eef50"
I20260619 06:48:52.082589 22127 raft_consensus.cc:3037] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:52.083590 21852 raft_consensus.cc:2454] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc0643f08674c3e8fa38ab673b06a96 in term 1.
I20260619 06:48:52.083591 22127 raft_consensus.cc:2454] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc0643f08674c3e8fa38ab673b06a96 in term 1.
I20260619 06:48:52.083778 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.079082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:45885 (local address 127.14.116.193:45409)
0619 06:48:52.079199 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.079212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:52.079294 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:52.079814 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:52.079817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:52.079835 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:52.079950 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:52.079960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.080870 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.080875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:52.082158 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:52.082168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.083333 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.083337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:52.083459 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:52.083479 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:48:52.083564 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:48:52.083654 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260619 06:48:52.083777 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.079348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:45409 (local address 127.14.116.194:45885)
0619 06:48:52.079476 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:52.079480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:52.079495 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:48:52.079540 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:52.079542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:52.079606 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:52.079698 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:52.080082 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.080768 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.082263 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.082932 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.083041 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:52.083089 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:52.083579 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 06:48:52.083667 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:52.083884 21925 leader_election.cc:302] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6324d8355f0b4cde941ec554834eef50, ecc0643f08674c3e8fa38ab673b06a96; no voters:
I20260619 06:48:52.084255 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.078831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:40643 (local address 127.14.116.193:34917)
0619 06:48:52.078935 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.078953 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:52.079062 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:52.080018 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:52.080024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:52.080047 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:52.080146 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:52.080154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.081077 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.081080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:52.082165 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:52.082170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.083863 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.083866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:52.083971 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:52.083998 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:48:52.084073 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 06:48:52.084144 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 06:48:52.084264 21989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385c826e6c784c2c87b5c3d216f56d4c" candidate_uuid: "890b226d9d0b460fbc14beb457ba427a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc0643f08674c3e8fa38ab673b06a96" is_pre_election: true
I20260619 06:48:52.084393 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.078933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:34917 (local address 127.14.116.195:40643)
0619 06:48:52.079056 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:52.079060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:52.079078 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:48:52.079698 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:52.079701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:52.079748 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:52.079823 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:52.080266 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.080965 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.082327 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.082931 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.083041 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:52.083089 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:52.084179 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0619 06:48:52.084275 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:48:52.084406 22280 raft_consensus.cc:2781] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:52.084602 22280 raft_consensus.cc:686] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 1 LEADER]: Becoming Leader. State: Replica: ecc0643f08674c3e8fa38ab673b06a96, State: Running, Role: LEADER
I20260619 06:48:52.084753 22127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385c826e6c784c2c87b5c3d216f56d4c" candidate_uuid: "890b226d9d0b460fbc14beb457ba427a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6324d8355f0b4cde941ec554834eef50" is_pre_election: true
I20260619 06:48:52.084743 22280 consensus_queue.cc:237] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } }
I20260619 06:48:52.084892 22127 raft_consensus.cc:2379] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 890b226d9d0b460fbc14beb457ba427a in current term 1: Already voted for candidate ecc0643f08674c3e8fa38ab673b06a96 in this term.
I20260619 06:48:52.085243 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.085003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:49177)
0619 06:48:52.085137 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.085186 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 06:48:52.086118 21788 leader_election.cc:302] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 890b226d9d0b460fbc14beb457ba427a; no voters: 6324d8355f0b4cde941ec554834eef50, ecc0643f08674c3e8fa38ab673b06a96
I20260619 06:48:52.086349 22282 raft_consensus.cc:2726] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 06:48:52.086486 21719 catalog_manager.cc:5495] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc0643f08674c3e8fa38ab673b06a96 (127.14.116.194). New cstate: current_term: 1 leader_uuid: "ecc0643f08674c3e8fa38ab673b06a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc0643f08674c3e8fa38ab673b06a96" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 45885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 } health_report { overall_health: UNKNOWN } } }
I20260619 06:48:52.152669 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.149940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.14.116.194:45885)
0619 06:48:52.150068 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:52.150072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:52.150086 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:52.150144 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:52.150146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:52.150187 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:52.150258 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:52.150465 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.150954 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.151749 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:52.151962 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:52.152038 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:52.152249 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:52.152313 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:52.152367 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:52.152494 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 06:48:52.152535 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:48:52.154680 21852 raft_consensus.cc:1261] T 385c826e6c784c2c87b5c3d216f56d4c P 890b226d9d0b460fbc14beb457ba427a [term 1 FOLLOWER]: Refusing update from remote peer ecc0643f08674c3e8fa38ab673b06a96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:52.154726 22127 raft_consensus.cc:1261] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 1 FOLLOWER]: Refusing update from remote peer ecc0643f08674c3e8fa38ab673b06a96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:52.155130 22280 consensus_queue.cc:1035] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "890b226d9d0b460fbc14beb457ba427a" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:52.155300 22289 consensus_queue.cc:1035] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6324d8355f0b4cde941ec554834eef50" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:52.156944 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.156678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.193:46881)
0619 06:48:52.156800 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.156865 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:48:52.157053 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.156540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:59400)
0619 06:48:52.156926 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.156980 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313}
I20260619 06:48:52.158066 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.157624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.0.0.1:59406)
0619 06:48:52.157942 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.157995 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245}
I20260619 06:48:52.158595 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.158328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.195:36231)
0619 06:48:52.158462 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.158513 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 06:48:52.160107 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.159751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:53839)
0619 06:48:52.159991 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.160038 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260619 06:48:52.161746 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.160797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.193:37735)
0619 06:48:52.161648 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.161691 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":792}
I20260619 06:48:52.163332 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.163113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.194:47433)
0619 06:48:52.163204 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.163258 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 06:48:52.164738 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.164351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40519 (local address 127.14.116.195:53871)
0619 06:48:52.164616 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.164663 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260619 06:48:52.165678 21717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:52.165939 21717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:52.169162 21717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b5a2dd1ea1040b09762a53247d70e05: Sending DeleteTablet for 3 replicas of tablet 385c826e6c784c2c87b5c3d216f56d4c
I20260619 06:48:52.169723 22107 tablet_service.cc:1514] Processing DeleteTablet for tablet 385c826e6c784c2c87b5c3d216f56d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:52 UTC) from {username='oryx', principal='oryx/127.14.116.252@KRBTEST.COM'} at 127.0.0.1:60376
I20260619 06:48:52.169790 21969 tablet_service.cc:1514] Processing DeleteTablet for tablet 385c826e6c784c2c87b5c3d216f56d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:52 UTC) from {username='oryx', principal='oryx/127.14.116.252@KRBTEST.COM'} at 127.0.0.1:32814
I20260619 06:48:52.170099 22301 tablet_replica.cc:330] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: stopping tablet replica
I20260619 06:48:52.170113 21832 tablet_service.cc:1514] Processing DeleteTablet for tablet 385c826e6c784c2c87b5c3d216f56d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:52 UTC) from {username='oryx', principal='oryx/127.14.116.252@KRBTEST.COM'} at 127.0.0.1:60456
I20260619 06:48:52.170277 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 21681
I20260619 06:48:52.170317 22301 raft_consensus.cc:2229] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:52.170912 22300 tablet_replica.cc:330] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: stopping tablet replica
I20260619 06:48:52.171034 22300 raft_consensus.cc:2229] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:52.171150 22300 raft_consensus.cc:2258] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:52.171181 22301 raft_consensus.cc:2258] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:52.171821 22301 ts_tablet_manager.cc:1905] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:52.172022 22300 ts_tablet_manager.cc:1905] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:52.173744 22301 ts_tablet_manager.cc:1918] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:52.173830 22301 log.cc:1196] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/385c826e6c784c2c87b5c3d216f56d4c
I20260619 06:48:52.174088 22301 ts_tablet_manager.cc:1939] T 385c826e6c784c2c87b5c3d216f56d4c P ecc0643f08674c3e8fa38ab673b06a96: Deleting consensus metadata
I20260619 06:48:52.174784 21706 catalog_manager.cc:4841] TS ecc0643f08674c3e8fa38ab673b06a96 (127.14.116.194:45885): tablet 385c826e6c784c2c87b5c3d216f56d4c (table test-table [id=bb1b5e560b544b0897723ced239481c3]) successfully deleted
I20260619 06:48:52.175006 22300 ts_tablet_manager.cc:1918] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:52.175089 22300 log.cc:1196] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/385c826e6c784c2c87b5c3d216f56d4c
I20260619 06:48:52.175392 22300 ts_tablet_manager.cc:1939] T 385c826e6c784c2c87b5c3d216f56d4c P 6324d8355f0b4cde941ec554834eef50: Deleting consensus metadata
I20260619 06:48:52.176045 21706 catalog_manager.cc:4841] TS 6324d8355f0b4cde941ec554834eef50 (127.14.116.195:40643): tablet 385c826e6c784c2c87b5c3d216f56d4c (table test-table [id=bb1b5e560b544b0897723ced239481c3]) successfully deleted
I20260619 06:48:52.179172 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 21906
I20260619 06:48:52.179368 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:52.179103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:40389 (local address 127.0.0.1:60466)
0619 06:48:52.179215 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:52.179286 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:48:52.179888 21706 catalog_manager.cc:4584] TS 890b226d9d0b460fbc14beb457ba427a (127.14.116.193:40389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 385c826e6c784c2c87b5c3d216f56d4c: Network error: Client connection negotiation failed: client connection to 127.14.116.193:40389: connect: Connection refused (error 111)
I20260619 06:48:52.186640 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 22042
I20260619 06:48:52.191944 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 21399
I20260619 06:48:52.197472 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 21509
2026-06-19T06:48:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7587 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:52 dist-test-slave-9x3b krb5kdc[22305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:52 dist-test-slave-9x3b krb5kdc[22305](info): set up 2 sockets
Jun 19 06:48:52 dist-test-slave-9x3b krb5kdc[22305](info): commencing operation
krb5kdc: starting...
W20260619 06:48:53.236249 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:53 dist-test-slave-9x3b krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.254.keytab.
Entry for principal oryx/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.254.keytab.
I20260619 06:48:53.347376 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:35567
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.254.keytab
--principal=oryx/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:53.452716 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:53.452950 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:53.453018 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:53.456943 22318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:53.456988 22318 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:48:53.457005 22318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:53.457022 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:53.457037 22318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:53.457052 22318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:53.462116 22318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:35567
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.254.keytab
--principal=oryx/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:53.463029 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:53.464061 22318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:48:53.472000 22318 server_base.cc:1034] running on GCE node
W20260619 06:48:53.472028 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:53.472044 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:53.472070 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:53.472793 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:53.473809 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:53.474960 22318 hybrid_clock.cc:648] HybridClock initialized: now 1781851733474946 us; error 25 us; skew 500 ppm
Jun 19 06:48:53 dist-test-slave-9x3b krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851733, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:53.477502 22318 init.cc:377] Logged in from keytab as oryx/127.14.116.254@KRBTEST.COM (short username oryx)
I20260619 06:48:53.478220 22318 webserver.cc:460] Webserver started at http://127.14.116.254:37057/ using document root <none> and password file <none>
I20260619 06:48:53.478688 22318 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:53.478742 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:53.478889 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:53.480751 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "72e82ffeaa3d42d18824511f9deacb3f"
format_stamp: "Formatted at 2026-06-19 06:48:53 on dist-test-slave-9x3b"
server_key: "64a8ef9abc32df46f69c240bf75caa99"
server_key_iv: "7d3c39543283eee574a32ab7d360e2b7"
server_key_version: "encryptionkey@0"
I20260619 06:48:53.481220 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "72e82ffeaa3d42d18824511f9deacb3f"
format_stamp: "Formatted at 2026-06-19 06:48:53 on dist-test-slave-9x3b"
server_key: "64a8ef9abc32df46f69c240bf75caa99"
server_key_iv: "7d3c39543283eee574a32ab7d360e2b7"
server_key_version: "encryptionkey@0"
I20260619 06:48:53.484639 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:48:53.486794 22333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:53.488081 22318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:53.488201 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "72e82ffeaa3d42d18824511f9deacb3f"
format_stamp: "Formatted at 2026-06-19 06:48:53 on dist-test-slave-9x3b"
server_key: "64a8ef9abc32df46f69c240bf75caa99"
server_key_iv: "7d3c39543283eee574a32ab7d360e2b7"
server_key_version: "encryptionkey@0"
I20260619 06:48:53.488314 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:53.503556 22318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:53.506134 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:53.506311 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:53.513589 22318 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:35567
I20260619 06:48:53.513597 22395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:35567 every 8 connection(s)
I20260619 06:48:53.514464 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:53.517041 22396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:53.522136 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f: Bootstrap starting.
I20260619 06:48:53.522789 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 22318
I20260619 06:48:53.522902 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:53.523169 14803 external_mini_cluster.cc:1428] Setting key 4e82c5b09618f56cdcb60e21dd7680b3
I20260619 06:48:53.524711 22396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:53.525389 22396 log.cc:824] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:53.527340 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f: No bootstrap required, opened a new log
I20260619 06:48:53.531042 22396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e82ffeaa3d42d18824511f9deacb3f" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35567 } }
I20260619 06:48:53.531286 22396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:53.531358 22396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e82ffeaa3d42d18824511f9deacb3f, State: Initialized, Role: FOLLOWER
Jun 19 06:48:53 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:53.531834 22396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e82ffeaa3d42d18824511f9deacb3f" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35567 } }
I20260619 06:48:53.532011 22396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:53.532079 22396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:53.532187 22396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:53.532999 22396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e82ffeaa3d42d18824511f9deacb3f" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35567 } }
I20260619 06:48:53.533365 22396 leader_election.cc:302] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72e82ffeaa3d42d18824511f9deacb3f; no voters:
I20260619 06:48:53.533704 22396 leader_election.cc:288] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:53.533793 22401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:53.534024 22401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [term 1 LEADER]: Becoming Leader. State: Replica: 72e82ffeaa3d42d18824511f9deacb3f, State: Running, Role: LEADER
I20260619 06:48:53.534366 22401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e82ffeaa3d42d18824511f9deacb3f" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35567 } }
I20260619 06:48:53.534994 22396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:53.536161 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72e82ffeaa3d42d18824511f9deacb3f. Latest consensus state: current_term: 1 leader_uuid: "72e82ffeaa3d42d18824511f9deacb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e82ffeaa3d42d18824511f9deacb3f" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35567 } } }
I20260619 06:48:53.536280 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:53.536558 22402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72e82ffeaa3d42d18824511f9deacb3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72e82ffeaa3d42d18824511f9deacb3f" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35567 } } }
I20260619 06:48:53.536648 22402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:53.537056 22407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:53.538012 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:53.524230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.14.116.254:35567)
0619 06:48:53.524596 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:53.524611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:53.524644 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:48:53.525260 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:53.525275 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:53.525625 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:53.525887 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:53.526608 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.528030 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.528838 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.529551 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.529705 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:53.532228 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:53.532232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:53.532262 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:53.532294 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:53.534289 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.534801 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.534806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.534811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.534936 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.535239 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.535242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.535243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.535728 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:53.536562 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:53.536711 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 06:48:53.537125 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":128,"threads_started":1}
I20260619 06:48:53.539978 22407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:53.545050 22407 catalog_manager.cc:1347] Generated new cluster ID: 40e5ec9f253d4639b5889eb52c8800aa
I20260619 06:48:53.545106 22407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:53.565574 22407 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:53.566835 22407 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:53.577975 22407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f: Generated new TSK 0
I20260619 06:48:53.578766 22407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab.
Entry for principal oryx/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab.
I20260619 06:48:53.612378 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35567
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab
--principal=oryx/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:53.718819 22422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:53.719045 22422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:53.719120 22422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:53.723140 22422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:53.723193 22422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:53.723230 22422 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:48:53.723320 22422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:53.728425 22422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab
--principal=oryx/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.254:35567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:53.729362 22422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:53.730312 22422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:53.739630 22428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:53.739573 22427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:53.739638 22430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:53.739799 22422 server_base.cc:1034] running on GCE node
I20260619 06:48:53.740245 22422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:53.740813 22422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:53.741961 22422 hybrid_clock.cc:648] HybridClock initialized: now 1781851733741947 us; error 25 us; skew 500 ppm
Jun 19 06:48:53 dist-test-slave-9x3b krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851733, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:53.744645 22422 init.cc:377] Logged in from keytab as oryx/127.14.116.193@KRBTEST.COM (short username oryx)
I20260619 06:48:53.745389 22422 webserver.cc:460] Webserver started at http://127.14.116.193:35211/ using document root <none> and password file <none>
I20260619 06:48:53.745874 22422 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:53.745929 22422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:53.746078 22422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:53.747996 22422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "9222f1e019294a17b45f1ca7fea91cb1"
format_stamp: "Formatted at 2026-06-19 06:48:53 on dist-test-slave-9x3b"
server_key: "c9d1a899d729f2a5012755559d3f6536"
server_key_iv: "250b4b0f012bef575a1cdb5cf53d407a"
server_key_version: "encryptionkey@0"
I20260619 06:48:53.748479 22422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "9222f1e019294a17b45f1ca7fea91cb1"
format_stamp: "Formatted at 2026-06-19 06:48:53 on dist-test-slave-9x3b"
server_key: "c9d1a899d729f2a5012755559d3f6536"
server_key_iv: "250b4b0f012bef575a1cdb5cf53d407a"
server_key_version: "encryptionkey@0"
I20260619 06:48:53.751813 22422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:53.754027 22437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:53.755142 22422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:53.755268 22422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "9222f1e019294a17b45f1ca7fea91cb1"
format_stamp: "Formatted at 2026-06-19 06:48:53 on dist-test-slave-9x3b"
server_key: "c9d1a899d729f2a5012755559d3f6536"
server_key_iv: "250b4b0f012bef575a1cdb5cf53d407a"
server_key_version: "encryptionkey@0"
I20260619 06:48:53.755369 22422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:53.769891 22422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:53.772987 22422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:53.773175 22422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:53.774505 22422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:53.774570 22422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:53.774643 22422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:53.774683 22422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:48:53 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851733, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:53.785038 22422 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:46355
I20260619 06:48:53.785250 22552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:46355 every 8 connection(s)
I20260619 06:48:53.785943 22422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:53.787460 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:53.775270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:34023 (local address 127.14.116.254:35567)
0619 06:48:53.775427 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:53.775432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:53.776252 (+ 820us) server_negotiation.cc:407] Connection header received
0619 06:48:53.777316 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:53.777321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:53.777385 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:53.777449 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:53.779143 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.779920 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.781054 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.781282 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.781398 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:53.784067 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:53.784072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:53.784076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:53.784105 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:53.785588 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.786063 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.786068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.786070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.786147 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.786523 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.786527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.786528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.786743 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:53.786821 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:53.787252 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 06:48:53.787343 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:48:53.787886 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 22422
I20260619 06:48:53.788034 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:53.788300 14803 external_mini_cluster.cc:1428] Setting key e3fb82b3fd03d88f2b0d7f7fb7154f1c
I20260619 06:48:53.788321 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:53.775680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35567 (local address 127.14.116.193:34023)
0619 06:48:53.776094 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:53.776141 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:53.777076 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:53.777861 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:53.777870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:53.778281 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:53.778961 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:53.778981 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.780035 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.780039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:53.780899 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:53.780907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.781448 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.781452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:53.781623 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:53.782180 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:53.782198 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:53.783911 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:53.785693 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:53.785699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:53.785711 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:53.785965 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:53.786240 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:53.786243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:53.786245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:53.786414 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:53.786840 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:53.786850 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:53.787056 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 06:48:53.787213 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 06:48:53.787438 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":124,"threads_started":1}
I20260619 06:48:53.789570 22553 heartbeater.cc:347] Connected to a master server at 127.14.116.254:35567
I20260619 06:48:53.791661 22553 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:53.792167 22553 heartbeater.cc:511] Master 127.14.116.254:35567 requested a full tablet report, sending...
I20260619 06:48:53.793650 22348 ts_manager.cc:194] Registered new tserver with Master: 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355)
I20260619 06:48:53.795253 22348 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:34023
I20260619 06:48:53.798761 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:53.791793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:39071 (local address 127.14.116.254:35567)
0619 06:48:53.791912 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:53.791918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:53.792163 (+ 245us) server_negotiation.cc:407] Connection header received
0619 06:48:53.792323 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:53.792329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:53.792380 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:53.792468 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:53.792887 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.793398 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.794281 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.794506 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.794657 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:53.795701 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:53.795705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:53.795708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:53.795731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:53.797130 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.797631 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.797637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.797641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.797726 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.798008 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.798011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.798013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.798173 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:53.798271 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:53.798511 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 06:48:53.798609 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":49}
I20260619 06:48:53.798774 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:53.791745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35567 (local address 127.14.116.193:39071)
0619 06:48:53.792080 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:53.792093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:53.792184 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:53.792512 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:53.792515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:53.792709 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:53.792809 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:53.792814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.793514 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.793517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:53.794148 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:53.794156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.794621 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.794625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:53.794714 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:53.795180 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:53.795197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:53.795549 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:53.797234 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:53.797240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:53.797242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:53.797512 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:53.797814 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:53.797819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:53.797820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:53.797909 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:53.798261 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:53.798264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:53.798356 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 06:48:53.798460 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:48:53.798597 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for oryx/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab.
Entry for principal oryx/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab.
I20260619 06:48:53.814739 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35567
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab
--principal=oryx/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:53.918952 22562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:53.919238 22562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:53.919281 22562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:53.923133 22562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:53.923177 22562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:53.923213 22562 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:48:53.923249 22562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:53.928125 22562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab
--principal=oryx/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.254:35567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:53.928975 22562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:53.929925 22562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:53.938750 22570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:53.938747 22568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:53.938789 22567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:53.938982 22562 server_base.cc:1034] running on GCE node
I20260619 06:48:53.939409 22562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:53.940007 22562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:53.941155 22562 hybrid_clock.cc:648] HybridClock initialized: now 1781851733941136 us; error 31 us; skew 500 ppm
Jun 19 06:48:53 dist-test-slave-9x3b krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851733, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:53.943703 22562 init.cc:377] Logged in from keytab as oryx/127.14.116.194@KRBTEST.COM (short username oryx)
I20260619 06:48:53.944515 22562 webserver.cc:460] Webserver started at http://127.14.116.194:41981/ using document root <none> and password file <none>
I20260619 06:48:53.945029 22562 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:53.945109 22562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:53.945302 22562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:53.947186 22562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "a5ddf23f63a94d92a71bab927a4ef9a0"
format_stamp: "Formatted at 2026-06-19 06:48:53 on dist-test-slave-9x3b"
server_key: "59d94129a05874b21e5f55356ccf8b4d"
server_key_iv: "6a230cae21b11a73339877678f4e6d6f"
server_key_version: "encryptionkey@0"
I20260619 06:48:53.947775 22562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "a5ddf23f63a94d92a71bab927a4ef9a0"
format_stamp: "Formatted at 2026-06-19 06:48:53 on dist-test-slave-9x3b"
server_key: "59d94129a05874b21e5f55356ccf8b4d"
server_key_iv: "6a230cae21b11a73339877678f4e6d6f"
server_key_version: "encryptionkey@0"
I20260619 06:48:53.951174 22562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:48:53.953377 22577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:53.954453 22562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:48:53.954571 22562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "a5ddf23f63a94d92a71bab927a4ef9a0"
format_stamp: "Formatted at 2026-06-19 06:48:53 on dist-test-slave-9x3b"
server_key: "59d94129a05874b21e5f55356ccf8b4d"
server_key_iv: "6a230cae21b11a73339877678f4e6d6f"
server_key_version: "encryptionkey@0"
I20260619 06:48:53.954668 22562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:53.968604 22562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:53.971213 22562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:53.971397 22562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:53.972808 22562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:53.972880 22562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:53.972947 22562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:53.972995 22562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:48:53 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851733, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:53.983256 22562 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:39901
I20260619 06:48:53.983536 22692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:39901 every 8 connection(s)
I20260619 06:48:53.984333 22562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:53.986016 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:53.973423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:44533 (local address 127.14.116.254:35567)
0619 06:48:53.973577 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:53.973584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:53.974818 (+ 1234us) server_negotiation.cc:407] Connection header received
0619 06:48:53.976010 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:53.976015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:53.976082 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:53.976188 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:53.977373 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.978253 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.979549 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.979987 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.980108 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:53.982458 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:53.982462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:53.982464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:53.982492 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:53.984103 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.984606 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.984611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.984616 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.984716 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.985119 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.985123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.985124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.985328 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:53.985417 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:53.985810 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 06:48:53.985901 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:48:53.986745 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:53.974241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35567 (local address 127.14.116.194:44533)
0619 06:48:53.974665 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:53.974711 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:53.975790 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:53.976345 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:53.976351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:53.976741 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:53.977241 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:53.977254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.978379 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.978385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:53.979450 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:53.979459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.980148 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.980154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:53.980337 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:53.980865 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:53.980881 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:53.982303 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:53.984200 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:53.984204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:53.984213 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:53.984515 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:53.984838 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:53.984841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:53.984843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:53.985007 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:53.985420 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:53.985424 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:53.985620 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 06:48:53.985766 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 06:48:53.985990 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":124,"threads_started":1}
I20260619 06:48:53.987934 22693 heartbeater.cc:347] Connected to a master server at 127.14.116.254:35567
I20260619 06:48:53.988071 22693 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:53.988538 22693 heartbeater.cc:511] Master 127.14.116.254:35567 requested a full tablet report, sending...
I20260619 06:48:53.989498 22348 ts_manager.cc:194] Registered new tserver with Master: a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901)
I20260619 06:48:53.990126 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 22562
I20260619 06:48:53.990239 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:53.990267 22348 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:44533
I20260619 06:48:53.990566 14803 external_mini_cluster.cc:1428] Setting key 73f36b038a725e9834757f1f46e5a167
I20260619 06:48:53.999527 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:53.989822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:46353 (local address 127.14.116.254:35567)
0619 06:48:53.989945 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:53.989949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:53.992668 (+ 2719us) server_negotiation.cc:407] Connection header received
0619 06:48:53.992800 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:53.992803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:53.992846 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:53.992909 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:53.993433 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.993898 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.995100 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.995292 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.995361 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:53.996629 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:53.996632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:53.996634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:53.996653 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:53.998001 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.998469 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.998471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.998473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.998536 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:53.998813 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:53.998815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:53.998816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:53.998995 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:53.999067 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:53.999340 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:48:53.999421 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:48:53.999744 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:53.989785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35567 (local address 127.14.116.194:46353)
0619 06:48:53.992583 (+ 2798us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:53.992601 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:53.992701 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:53.992955 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:53.992961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:53.993186 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:53.993332 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:53.993342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.994015 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.994020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:53.994993 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:53.995006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:53.995391 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:53.995397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:53.995536 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:53.996019 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:53.996037 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:53.996512 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:53.998109 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:53.998112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:53.998117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:53.998367 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:53.998620 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:53.998624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:53.998629 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:53.998731 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:53.999089 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:53.999093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:53.999183 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 06:48:53.999306 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 06:48:53.999459 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2696,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for oryx/127.14.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab.
Entry for principal oryx/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab.
I20260619 06:48:54.016052 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35567
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab
--principal=oryx/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:54.123570 22702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:54.123895 22702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:54.123965 22702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:54.127895 22702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:54.127964 22702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:54.128026 22702 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:48:54.128100 22702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:54.133491 22702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab
--principal=oryx/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.116.254:35567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:54.134718 22702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:54.135995 22702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:54.145440 22708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:54.145448 22710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:54.145653 22702 server_base.cc:1034] running on GCE node
W20260619 06:48:54.145448 22707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:54.146133 22702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:54.146754 22702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:54.147907 22702 hybrid_clock.cc:648] HybridClock initialized: now 1781851734147892 us; error 25 us; skew 500 ppm
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851734, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:54.150472 22702 init.cc:377] Logged in from keytab as oryx/127.14.116.195@KRBTEST.COM (short username oryx)
I20260619 06:48:54.151331 22702 webserver.cc:460] Webserver started at http://127.14.116.195:33649/ using document root <none> and password file <none>
I20260619 06:48:54.151928 22702 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:54.151990 22702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:54.152212 22702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:54.154856 22702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "bfa5e57e1c354da39237cfb57987017d"
format_stamp: "Formatted at 2026-06-19 06:48:54 on dist-test-slave-9x3b"
server_key: "155ad906db91ff1e9b45a6130972f3e3"
server_key_iv: "ded65b6c7e232c92949cfc66588946be"
server_key_version: "encryptionkey@0"
I20260619 06:48:54.155411 22702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "bfa5e57e1c354da39237cfb57987017d"
format_stamp: "Formatted at 2026-06-19 06:48:54 on dist-test-slave-9x3b"
server_key: "155ad906db91ff1e9b45a6130972f3e3"
server_key_iv: "ded65b6c7e232c92949cfc66588946be"
server_key_version: "encryptionkey@0"
I20260619 06:48:54.159178 22702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:48:54.161541 22717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:54.162642 22702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:54.162755 22702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "bfa5e57e1c354da39237cfb57987017d"
format_stamp: "Formatted at 2026-06-19 06:48:54 on dist-test-slave-9x3b"
server_key: "155ad906db91ff1e9b45a6130972f3e3"
server_key_iv: "ded65b6c7e232c92949cfc66588946be"
server_key_version: "encryptionkey@0"
I20260619 06:48:54.162853 22702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:54.185019 22702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:54.188345 22702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:54.188542 22702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:54.190068 22702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:54.190141 22702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:54.190208 22702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:54.190248 22702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851734, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:54.201470 22702 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:40975
I20260619 06:48:54.201606 22832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:40975 every 8 connection(s)
I20260619 06:48:54.202528 22702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:54.204162 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.190764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:58219 (local address 127.14.116.254:35567)
0619 06:48:54.190936 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.190942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.191984 (+ 1042us) server_negotiation.cc:407] Connection header received
0619 06:48:54.193122 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.193126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.193198 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.193276 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:54.194771 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.195787 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.196953 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.197190 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.197308 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.200413 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:54.200416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:54.200419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:54.200447 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:54.202151 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:54.202629 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:54.202633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:54.202634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:54.202722 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:54.203114 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:54.203117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:54.203119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:54.203348 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:54.203432 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.203954 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.204052 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:48:54.204895 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.191238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35567 (local address 127.14.116.195:58219)
0619 06:48:54.191812 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.191867 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.192852 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.193607 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.193614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.193950 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:54.194594 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.194621 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.195916 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.195919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.196821 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.196830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.197359 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.197363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.197559 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.198177 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:54.198213 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:54.200266 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:54.202260 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:54.202264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:54.202274 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:54.202521 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:54.202824 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:54.202826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:54.202827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:54.203000 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:54.203448 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:54.203452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:54.203733 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.203889 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.204111 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":207,"threads_started":1}
I20260619 06:48:54.206136 22833 heartbeater.cc:347] Connected to a master server at 127.14.116.254:35567
I20260619 06:48:54.206339 22833 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:54.206781 22833 heartbeater.cc:511] Master 127.14.116.254:35567 requested a full tablet report, sending...
I20260619 06:48:54.207765 22349 ts_manager.cc:194] Registered new tserver with Master: bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975)
I20260619 06:48:54.208315 22349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:58219
I20260619 06:48:54.211694 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 22702
I20260619 06:48:54.211812 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:54.212091 14803 external_mini_cluster.cc:1428] Setting key 3f70f32cf1bbd534b16f8c392358d9c9
I20260619 06:48:54.214156 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:48:54.215451 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.208390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35567 (local address 127.14.116.195:55929)
0619 06:48:54.208763 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.208778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.208872 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.209110 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.209112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.209311 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:54.209428 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.209439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.210153 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.210156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.210708 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.210715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.211162 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.211165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.211257 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.211756 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:54.211768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:54.212137 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:54.213997 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:54.214000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:54.214002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:54.214268 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:54.214566 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:54.214569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:54.214570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:54.214642 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:54.215044 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:54.215046 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:54.215118 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.215198 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.215318 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260619 06:48:54.215677 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.208498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:55929 (local address 127.14.116.254:35567)
0619 06:48:54.208662 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.208668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.208848 (+ 180us) server_negotiation.cc:407] Connection header received
0619 06:48:54.208966 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.208969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.209014 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.209096 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:54.209542 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.210032 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.210830 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.211044 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.211161 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.212281 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:54.212285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:54.212288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:54.212310 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:54.213891 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:54.214387 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:54.214390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:54.214392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:54.214480 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:54.214732 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:54.214736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:54.214739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:54.214948 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:54.215053 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.215279 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.215404 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:48:54.239468 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:54.233106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.14.116.254:35567)
0619 06:48:54.233246 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.233250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.233262 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:48:54.233299 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.233301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.233338 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.233410 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:54.234068 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.234591 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.235585 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.235847 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.235942 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.239286 (+ 3344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47872: BlockingRecv error: recv got EOF from 127.0.0.1:47872 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:54.248085 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.240829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.14.116.254:35567)
0619 06:48:54.240958 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.240961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.241021 (+ 60us) server_negotiation.cc:407] Connection header received
0619 06:48:54.241096 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.241098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.241130 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.241196 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:54.241702 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.242153 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.242872 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.243063 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.243133 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.245311 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:54.245314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:54.245315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:54.245335 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:54.246564 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:54.247006 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:54.247009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:54.247010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:54.247079 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:54.247317 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:54.247319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:54.247320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:54.247469 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:54.247567 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.247847 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.247967 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:48:54.250699 22349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:54.253640 22349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:54.267838 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.262141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.14.116.195:40975)
0619 06:48:54.263382 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.263393 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.263428 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:48:54.263498 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.263503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.263709 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.263849 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:54.264035 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.264831 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.266296 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.267131 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.267263 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.267384 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.267575 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.267694 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1125,"thread_start_us":135,"threads_started":1}
I20260619 06:48:54.267865 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.262019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:40975 (local address 127.0.0.1:59002)
0619 06:48:54.263024 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.263040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.263126 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.263836 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.263839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.263857 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:54.263942 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.263950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.264937 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.264944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.266175 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.266184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.267236 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.267240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.267357 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.267399 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.267506 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.267727 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":68,"threads_started":1}
I20260619 06:48:54.268090 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.262095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:39901 (local address 127.0.0.1:49240)
0619 06:48:54.262928 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.262975 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.263091 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.263657 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.263662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.263684 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:54.263797 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.263804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.265217 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.265220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.266523 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.266535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.267653 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.267657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.267772 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.267794 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.267859 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.267939 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":19,"thread_start_us":80,"threads_started":1}
I20260619 06:48:54.268143 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.262735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.14.116.193:46355)
0619 06:48:54.263500 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.263512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.263529 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:48:54.263617 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.263640 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.263829 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.263965 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:54.264282 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.265304 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.266674 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.267250 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.267375 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.267490 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.267932 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.268012 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":88,"threads_started":1}
I20260619 06:48:54.268378 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.262349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46355 (local address 127.0.0.1:46050)
0619 06:48:54.263281 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.263290 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.263339 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.264073 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.264078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.264096 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:54.264165 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.264170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.265412 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.265415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.266537 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.266543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.267547 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.267550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.267644 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.267689 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.267764 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.267823 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260619 06:48:54.268832 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.262747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.14.116.194:39901)
0619 06:48:54.263124 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.263137 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.263164 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:48:54.263241 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.263246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.263481 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.263649 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:54.263939 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.265069 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.266636 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.267419 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.268432 (+ 1013us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.268563 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.268605 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.268692 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":91,"threads_started":1}
I20260619 06:48:54.270874 22627 tablet_service.cc:1467] Processing CreateTablet for tablet c59156394d4f43a1988a5f29ef5144df (DEFAULT_TABLE table=test-table [id=6656e3992c634953a63a991d1edcd5e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:54.270901 22767 tablet_service.cc:1467] Processing CreateTablet for tablet c59156394d4f43a1988a5f29ef5144df (DEFAULT_TABLE table=test-table [id=6656e3992c634953a63a991d1edcd5e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:54.270901 22487 tablet_service.cc:1467] Processing CreateTablet for tablet c59156394d4f43a1988a5f29ef5144df (DEFAULT_TABLE table=test-table [id=6656e3992c634953a63a991d1edcd5e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:54.271865 22627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59156394d4f43a1988a5f29ef5144df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:54.271879 22767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59156394d4f43a1988a5f29ef5144df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:54.271874 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59156394d4f43a1988a5f29ef5144df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:54.277886 22861 tablet_bootstrap.cc:492] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: Bootstrap starting.
I20260619 06:48:54.279466 22862 tablet_bootstrap.cc:492] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: Bootstrap starting.
I20260619 06:48:54.279466 22863 tablet_bootstrap.cc:492] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: Bootstrap starting.
I20260619 06:48:54.280117 22861 tablet_bootstrap.cc:654] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:54.281032 22861 log.cc:824] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:54.281531 22863 tablet_bootstrap.cc:654] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:54.281531 22862 tablet_bootstrap.cc:654] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:54.282335 22862 log.cc:824] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:54.282335 22863 log.cc:824] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:54.282819 22861 tablet_bootstrap.cc:492] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: No bootstrap required, opened a new log
I20260619 06:48:54.283015 22861 ts_tablet_manager.cc:1397] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:48:54.284219 22862 tablet_bootstrap.cc:492] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: No bootstrap required, opened a new log
I20260619 06:48:54.284219 22863 tablet_bootstrap.cc:492] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: No bootstrap required, opened a new log
I20260619 06:48:54.284389 22863 ts_tablet_manager.cc:1397] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 06:48:54.284389 22862 ts_tablet_manager.cc:1397] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 06:48:54.286003 22861 raft_consensus.cc:348] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } }
I20260619 06:48:54.286257 22861 raft_consensus.cc:374] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:54.286314 22861 raft_consensus.cc:729] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ddf23f63a94d92a71bab927a4ef9a0, State: Initialized, Role: FOLLOWER
I20260619 06:48:54.286742 22861 consensus_queue.cc:260] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } }
I20260619 06:48:54.287506 22693 heartbeater.cc:503] Master 127.14.116.254:35567 was elected leader, sending a full tablet report...
I20260619 06:48:54.287792 22861 ts_tablet_manager.cc:1428] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:48:54.288172 22863 raft_consensus.cc:348] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } }
I20260619 06:48:54.288460 22863 raft_consensus.cc:374] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:54.288527 22863 raft_consensus.cc:729] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9222f1e019294a17b45f1ca7fea91cb1, State: Initialized, Role: FOLLOWER
I20260619 06:48:54.288751 22862 raft_consensus.cc:348] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } }
I20260619 06:48:54.288996 22862 raft_consensus.cc:374] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:54.289049 22862 raft_consensus.cc:729] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa5e57e1c354da39237cfb57987017d, State: Initialized, Role: FOLLOWER
I20260619 06:48:54.289098 22863 consensus_queue.cc:260] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } }
I20260619 06:48:54.289474 22862 consensus_queue.cc:260] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } }
I20260619 06:48:54.290033 22863 ts_tablet_manager.cc:1428] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 06:48:54.290107 22833 heartbeater.cc:503] Master 127.14.116.254:35567 was elected leader, sending a full tablet report...
I20260619 06:48:54.290323 22553 heartbeater.cc:503] Master 127.14.116.254:35567 was elected leader, sending a full tablet report...
I20260619 06:48:54.290414 22862 ts_tablet_manager.cc:1428] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260619 06:48:54.349398 22868 raft_consensus.cc:482] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:54.349601 22868 raft_consensus.cc:504] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } }
I20260619 06:48:54.350672 22868 leader_election.cc:288] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975), a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901)
I20260619 06:48:54.354336 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.350935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:48519 (local address 127.14.116.195:40975)
0619 06:48:54.351061 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.351065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.351229 (+ 164us) server_negotiation.cc:407] Connection header received
0619 06:48:54.351389 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.351393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.351449 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.351531 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:54.351802 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.352299 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.353390 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.353773 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.353891 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.353944 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.354138 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.354199 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:48:54.354359 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.350842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:40975 (local address 127.14.116.193:48519)
0619 06:48:54.351157 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.351171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.351283 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.351542 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.351545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.351559 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:54.351696 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.351701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.352404 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.352409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.353264 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.353272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.353891 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.353895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.354013 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.354041 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.354125 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.354193 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":47,"thread_start_us":87,"threads_started":1}
I20260619 06:48:54.354382 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.350892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:39901 (local address 127.14.116.193:40023)
0619 06:48:54.351197 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.351208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.351316 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.351591 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.351594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.351631 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:54.351739 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.351744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.352458 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.352463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.353266 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.353274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.353902 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.353906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.354057 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.354073 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.354156 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.354220 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":60,"thread_start_us":76,"threads_started":1}
I20260619 06:48:54.354363 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.351041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:40023 (local address 127.14.116.194:39901)
0619 06:48:54.351182 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.351190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.351277 (+ 87us) server_negotiation.cc:407] Connection header received
0619 06:48:54.351426 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.351433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.351473 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.351558 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:54.351841 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.352340 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.353389 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.353777 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.353902 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.353956 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.354169 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.354248 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:48:54.354888 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59156394d4f43a1988a5f29ef5144df" candidate_uuid: "9222f1e019294a17b45f1ca7fea91cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa5e57e1c354da39237cfb57987017d" is_pre_election: true
I20260619 06:48:54.354892 22647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59156394d4f43a1988a5f29ef5144df" candidate_uuid: "9222f1e019294a17b45f1ca7fea91cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" is_pre_election: true
I20260619 06:48:54.355152 22787 raft_consensus.cc:2454] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9222f1e019294a17b45f1ca7fea91cb1 in term 0.
I20260619 06:48:54.355176 22647 raft_consensus.cc:2454] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9222f1e019294a17b45f1ca7fea91cb1 in term 0.
I20260619 06:48:54.355592 22439 leader_election.cc:302] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9222f1e019294a17b45f1ca7fea91cb1, bfa5e57e1c354da39237cfb57987017d; no voters:
I20260619 06:48:54.355881 22868 raft_consensus.cc:2781] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:54.355976 22868 raft_consensus.cc:482] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:54.356034 22868 raft_consensus.cc:3037] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:54.356998 22868 raft_consensus.cc:504] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } }
I20260619 06:48:54.357321 22868 leader_election.cc:288] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [CANDIDATE]: Term 1 election: Requested vote from peers bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975), a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901)
I20260619 06:48:54.357658 22647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59156394d4f43a1988a5f29ef5144df" candidate_uuid: "9222f1e019294a17b45f1ca7fea91cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0"
I20260619 06:48:54.357664 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59156394d4f43a1988a5f29ef5144df" candidate_uuid: "9222f1e019294a17b45f1ca7fea91cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa5e57e1c354da39237cfb57987017d"
I20260619 06:48:54.357774 22787 raft_consensus.cc:3037] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:54.357774 22647 raft_consensus.cc:3037] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:54.359131 22787 raft_consensus.cc:2454] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9222f1e019294a17b45f1ca7fea91cb1 in term 1.
I20260619 06:48:54.359397 22439 leader_election.cc:302] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9222f1e019294a17b45f1ca7fea91cb1, bfa5e57e1c354da39237cfb57987017d; no voters:
I20260619 06:48:54.359565 22868 raft_consensus.cc:2781] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:54.359553 22647 raft_consensus.cc:2454] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9222f1e019294a17b45f1ca7fea91cb1 in term 1.
I20260619 06:48:54.359807 22868 raft_consensus.cc:686] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 LEADER]: Becoming Leader. State: Replica: 9222f1e019294a17b45f1ca7fea91cb1, State: Running, Role: LEADER
I20260619 06:48:54.360113 22868 consensus_queue.cc:237] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } }
I20260619 06:48:54.362974 22349 catalog_manager.cc:5495] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193). New cstate: current_term: 1 leader_uuid: "9222f1e019294a17b45f1ca7fea91cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } health_report { overall_health: HEALTHY } } }
I20260619 06:48:54.390411 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.387116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.14.116.193:46355)
0619 06:48:54.387232 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.387236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.387247 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:48:54.387313 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.387315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.387363 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.387511 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:54.387768 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.388252 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.389106 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.389311 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.389387 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.389674 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:54.390115 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:54.390200 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.390261 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.390307 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:54.395998 22647 raft_consensus.cc:1261] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Refusing update from remote peer 9222f1e019294a17b45f1ca7fea91cb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:54.396366 22787 raft_consensus.cc:1261] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Refusing update from remote peer 9222f1e019294a17b45f1ca7fea91cb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:54.396711 22872 consensus_queue.cc:1035] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:54.396953 22868 consensus_queue.cc:1035] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:54.404054 22875 mvcc.cc:205] Tried to move back new op lower bound from 7298464704079654912 to 7298464703941763072. Current Snapshot: MvccSnapshot[applied={T|T < 7298464704079654912}]
I20260619 06:48:54.405588 22876 mvcc.cc:205] Tried to move back new op lower bound from 7298464704079654912 to 7298464703941763072. Current Snapshot: MvccSnapshot[applied={T|T < 7298464704079654912}]
I20260619 06:48:54.405614 22877 mvcc.cc:205] Tried to move back new op lower bound from 7298464704079654912 to 7298464703941763072. Current Snapshot: MvccSnapshot[applied={T|T < 7298464704079654912}]
I20260619 06:48:54.412000 22350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:54.412135 22350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:54.414362 22350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f: Sending DeleteTablet for 3 replicas of tablet c59156394d4f43a1988a5f29ef5144df
I20260619 06:48:54.415040 22627 tablet_service.cc:1514] Processing DeleteTablet for tablet c59156394d4f43a1988a5f29ef5144df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:54 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:49240
I20260619 06:48:54.415040 22767 tablet_service.cc:1514] Processing DeleteTablet for tablet c59156394d4f43a1988a5f29ef5144df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:54 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:59002
I20260619 06:48:54.415108 22487 tablet_service.cc:1514] Processing DeleteTablet for tablet c59156394d4f43a1988a5f29ef5144df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:54 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:46050
I20260619 06:48:54.415495 22884 tablet_replica.cc:330] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: stopping tablet replica
I20260619 06:48:54.415558 22886 tablet_replica.cc:330] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: stopping tablet replica
I20260619 06:48:54.415870 22884 raft_consensus.cc:2229] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:54.415871 22886 raft_consensus.cc:2229] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:54.416056 22350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:48:54.416217 22886 raft_consensus.cc:2258] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:54.416406 22884 raft_consensus.cc:2258] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 06:48:54.416460 22350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:54.416781 22885 tablet_replica.cc:330] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: stopping tablet replica
I20260619 06:48:54.417022 22885 raft_consensus.cc:2229] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:54.417256 22885 raft_consensus.cc:2258] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:54.418123 22884 ts_tablet_manager.cc:1905] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:54.418545 22885 ts_tablet_manager.cc:1905] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:54.421025 22884 ts_tablet_manager.cc:1918] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:54.421096 22885 ts_tablet_manager.cc:1918] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:54.421190 22885 log.cc:1196] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/c59156394d4f43a1988a5f29ef5144df
I20260619 06:48:54.421304 22884 log.cc:1196] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/c59156394d4f43a1988a5f29ef5144df
I20260619 06:48:54.421497 22885 ts_tablet_manager.cc:1939] T c59156394d4f43a1988a5f29ef5144df P bfa5e57e1c354da39237cfb57987017d: Deleting consensus metadata
I20260619 06:48:54.421772 22884 ts_tablet_manager.cc:1939] T c59156394d4f43a1988a5f29ef5144df P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting consensus metadata
I20260619 06:48:54.422026 22487 tablet_service.cc:1467] Processing CreateTablet for tablet c63aebd0f1864f9fb896db480ef64136 (DEFAULT_TABLE table=test-table [id=7e36c568b1834e33b3772e0db53841db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:54.422287 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63aebd0f1864f9fb896db480ef64136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:54.422617 22335 catalog_manager.cc:4841] TS bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975): tablet c59156394d4f43a1988a5f29ef5144df (table test-table [id=6656e3992c634953a63a991d1edcd5e5]) successfully deleted
I20260619 06:48:54.423867 22767 tablet_service.cc:1467] Processing CreateTablet for tablet c63aebd0f1864f9fb896db480ef64136 (DEFAULT_TABLE table=test-table [id=7e36c568b1834e33b3772e0db53841db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:54.424104 22767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63aebd0f1864f9fb896db480ef64136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:54.426165 22862 tablet_bootstrap.cc:492] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: Bootstrap starting.
I20260619 06:48:54.426326 22334 catalog_manager.cc:4841] TS a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901): tablet c59156394d4f43a1988a5f29ef5144df (table test-table [id=6656e3992c634953a63a991d1edcd5e5]) successfully deleted
I20260619 06:48:54.426890 22627 tablet_service.cc:1467] Processing CreateTablet for tablet c63aebd0f1864f9fb896db480ef64136 (DEFAULT_TABLE table=test-table [id=7e36c568b1834e33b3772e0db53841db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:54.427098 22862 tablet_bootstrap.cc:654] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:54.427304 22627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63aebd0f1864f9fb896db480ef64136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:54.428996 22863 tablet_bootstrap.cc:492] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: Bootstrap starting.
I20260619 06:48:54.429602 22861 tablet_bootstrap.cc:492] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: Bootstrap starting.
I20260619 06:48:54.429960 22863 tablet_bootstrap.cc:654] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:54.430565 22861 tablet_bootstrap.cc:654] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:54.431247 22863 tablet_bootstrap.cc:492] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: No bootstrap required, opened a new log
I20260619 06:48:54.431330 22863 ts_tablet_manager.cc:1397] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 06:48:54.431804 22863 raft_consensus.cc:348] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.431924 22863 raft_consensus.cc:374] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:54.431958 22863 raft_consensus.cc:729] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9222f1e019294a17b45f1ca7fea91cb1, State: Initialized, Role: FOLLOWER
I20260619 06:48:54.432118 22861 tablet_bootstrap.cc:492] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: No bootstrap required, opened a new log
I20260619 06:48:54.432075 22863 consensus_queue.cc:260] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.432181 22861 ts_tablet_manager.cc:1397] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:48:54.432359 22863 ts_tablet_manager.cc:1428] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:48:54.432619 22861 raft_consensus.cc:348] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.432718 22861 raft_consensus.cc:374] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:54.432760 22861 raft_consensus.cc:729] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ddf23f63a94d92a71bab927a4ef9a0, State: Initialized, Role: FOLLOWER
I20260619 06:48:54.432879 22861 consensus_queue.cc:260] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.433116 22861 ts_tablet_manager.cc:1428] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:54.436659 22862 tablet_bootstrap.cc:492] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: No bootstrap required, opened a new log
I20260619 06:48:54.436764 22862 ts_tablet_manager.cc:1397] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20260619 06:48:54.437294 22862 raft_consensus.cc:348] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.437400 22862 raft_consensus.cc:374] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:54.437434 22862 raft_consensus.cc:729] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa5e57e1c354da39237cfb57987017d, State: Initialized, Role: FOLLOWER
I20260619 06:48:54.437552 22862 consensus_queue.cc:260] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.437780 22862 ts_tablet_manager.cc:1428] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:48:54.438611 22886 ts_tablet_manager.cc:1905] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:54.441203 22886 ts_tablet_manager.cc:1918] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:54.441309 22886 log.cc:1196] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/c59156394d4f43a1988a5f29ef5144df
I20260619 06:48:54.441640 22886 ts_tablet_manager.cc:1939] T c59156394d4f43a1988a5f29ef5144df P 9222f1e019294a17b45f1ca7fea91cb1: Deleting consensus metadata
I20260619 06:48:54.442646 22337 catalog_manager.cc:4841] TS 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355): tablet c59156394d4f43a1988a5f29ef5144df (table test-table [id=6656e3992c634953a63a991d1edcd5e5]) successfully deleted
W20260619 06:48:54.453450 22834 tablet.cc:2369] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:48:54.485483 22694 tablet.cc:2369] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:48:54.537153 22554 tablet.cc:2369] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:48:54.629746 22869 raft_consensus.cc:482] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:54.629915 22869 raft_consensus.cc:504] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.630985 22869 leader_election.cc:288] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355), a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901)
I20260619 06:48:54.635291 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.631245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:39901 (local address 127.14.116.195:33245)
0619 06:48:54.631352 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.631367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.631467 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.631771 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.631774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.631796 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:54.631938 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.631945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.632906 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.632908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.634115 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.634125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.634890 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.634893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.634971 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.634991 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.635051 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.635135 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 06:48:54.635322 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.631361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:33245 (local address 127.14.116.194:39901)
0619 06:48:54.631478 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.631504 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.631522 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:48:54.631578 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.631581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.631657 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.631739 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:54.632049 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.632802 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.634245 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.634785 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.634897 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.634959 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.635096 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.635171 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:48:54.635798 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.631353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:41215 (local address 127.14.116.193:46355)
0619 06:48:54.631486 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.631490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.631932 (+ 442us) server_negotiation.cc:407] Connection header received
0619 06:48:54.632058 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.632061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.632111 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.632186 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:54.632410 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.633060 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.634402 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.635015 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.635143 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.635187 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.635552 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.635695 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:48:54.635761 22647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63aebd0f1864f9fb896db480ef64136" candidate_uuid: "bfa5e57e1c354da39237cfb57987017d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" is_pre_election: true
I20260619 06:48:54.635910 22647 raft_consensus.cc:2454] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa5e57e1c354da39237cfb57987017d in term 0.
I20260619 06:48:54.636224 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.631245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46355 (local address 127.14.116.195:41215)
0619 06:48:54.631819 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.631832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.631940 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.632201 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.632203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.632216 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:54.632305 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.632308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.633151 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.633154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.634302 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.634309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.635158 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.635162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.635269 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.635290 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.635931 (+ 641us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.636005 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":295,"thread_start_us":97,"threads_started":1}
I20260619 06:48:54.636293 22718 leader_election.cc:302] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ddf23f63a94d92a71bab927a4ef9a0, bfa5e57e1c354da39237cfb57987017d; no voters:
I20260619 06:48:54.636545 22869 raft_consensus.cc:2781] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:54.636616 22869 raft_consensus.cc:482] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:54.636655 22869 raft_consensus.cc:3037] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:54.636716 22507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63aebd0f1864f9fb896db480ef64136" candidate_uuid: "bfa5e57e1c354da39237cfb57987017d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9222f1e019294a17b45f1ca7fea91cb1" is_pre_election: true
I20260619 06:48:54.636935 22507 raft_consensus.cc:2454] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa5e57e1c354da39237cfb57987017d in term 0.
I20260619 06:48:54.637815 22869 raft_consensus.cc:504] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.638139 22869 leader_election.cc:288] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [CANDIDATE]: Term 1 election: Requested vote from peers 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355), a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901)
I20260619 06:48:54.638401 22507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63aebd0f1864f9fb896db480ef64136" candidate_uuid: "bfa5e57e1c354da39237cfb57987017d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9222f1e019294a17b45f1ca7fea91cb1"
I20260619 06:48:54.638468 22647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63aebd0f1864f9fb896db480ef64136" candidate_uuid: "bfa5e57e1c354da39237cfb57987017d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0"
I20260619 06:48:54.638518 22507 raft_consensus.cc:3037] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:54.638599 22647 raft_consensus.cc:3037] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:54.639657 22647 raft_consensus.cc:2454] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa5e57e1c354da39237cfb57987017d in term 1.
I20260619 06:48:54.639662 22507 raft_consensus.cc:2454] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa5e57e1c354da39237cfb57987017d in term 1.
I20260619 06:48:54.639912 22718 leader_election.cc:302] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ddf23f63a94d92a71bab927a4ef9a0, bfa5e57e1c354da39237cfb57987017d; no voters:
I20260619 06:48:54.640108 22869 raft_consensus.cc:2781] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:54.640311 22869 raft_consensus.cc:686] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 1 LEADER]: Becoming Leader. State: Replica: bfa5e57e1c354da39237cfb57987017d, State: Running, Role: LEADER
I20260619 06:48:54.640558 22869 consensus_queue.cc:237] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.642541 22349 catalog_manager.cc:5495] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d reported cstate change: term changed from 0 to 1, leader changed from <none> to bfa5e57e1c354da39237cfb57987017d (127.14.116.195). New cstate: current_term: 1 leader_uuid: "bfa5e57e1c354da39237cfb57987017d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } health_report { overall_health: HEALTHY } } }
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851733, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.254@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:54.697273 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.689336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35567 (local address 127.0.0.1:47884)
0619 06:48:54.689588 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.689599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.689723 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.689972 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.689975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.690514 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:54.690612 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.690619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.691364 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.691371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.692016 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.692024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.692457 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.692460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.692529 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.693113 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:54.693124 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:54.694493 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:54.695988 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:54.695996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:54.696001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:54.696214 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:54.696444 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:54.696446 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:54.696447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:54.696518 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:54.696827 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:54.696831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:54.696932 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.697020 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.697119 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260619 06:48:54.697279 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.689382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.14.116.254:35567)
0619 06:48:54.689506 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.689511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.689670 (+ 159us) server_negotiation.cc:407] Connection header received
0619 06:48:54.689822 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.689826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.689868 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.689957 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:54.690724 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.691264 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.692125 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.692339 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.692458 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.694619 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:54.694621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:54.694623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:54.694641 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:54.695895 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:54.696295 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:54.696298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:54.696299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:54.696365 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:54.696590 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:54.696592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:54.696593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:54.696741 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:54.696812 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.697076 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.697160 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:48:54.700825 22349 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:47884:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:48:54.701537 22349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:54.705917 22627 tablet_service.cc:1467] Processing CreateTablet for tablet 347643bb76c04afc980a7628f2e9e066 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5feb8ef7bfe24dc685f672dc5f5a3b99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:48:54.705917 22487 tablet_service.cc:1467] Processing CreateTablet for tablet 347643bb76c04afc980a7628f2e9e066 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5feb8ef7bfe24dc685f672dc5f5a3b99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:48:54.706126 22767 tablet_service.cc:1467] Processing CreateTablet for tablet 347643bb76c04afc980a7628f2e9e066 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5feb8ef7bfe24dc685f672dc5f5a3b99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:48:54.706290 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347643bb76c04afc980a7628f2e9e066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:54.706290 22627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347643bb76c04afc980a7628f2e9e066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:54.706522 22767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347643bb76c04afc980a7628f2e9e066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:54.708909 22862 tablet_bootstrap.cc:492] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: Bootstrap starting.
I20260619 06:48:54.708938 22863 tablet_bootstrap.cc:492] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: Bootstrap starting.
I20260619 06:48:54.709012 22861 tablet_bootstrap.cc:492] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: Bootstrap starting.
I20260619 06:48:54.709800 22863 tablet_bootstrap.cc:654] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:54.709860 22862 tablet_bootstrap.cc:654] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:54.709860 22861 tablet_bootstrap.cc:654] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:54.711077 22863 tablet_bootstrap.cc:492] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: No bootstrap required, opened a new log
I20260619 06:48:54.711110 22861 tablet_bootstrap.cc:492] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: No bootstrap required, opened a new log
I20260619 06:48:54.711163 22863 ts_tablet_manager.cc:1397] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:54.711207 22861 ts_tablet_manager.cc:1397] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:54.711359 22862 tablet_bootstrap.cc:492] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: No bootstrap required, opened a new log
I20260619 06:48:54.711445 22862 ts_tablet_manager.cc:1397] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 06:48:54.711578 22863 raft_consensus.cc:348] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.711709 22863 raft_consensus.cc:374] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:54.711750 22863 raft_consensus.cc:729] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9222f1e019294a17b45f1ca7fea91cb1, State: Initialized, Role: FOLLOWER
I20260619 06:48:54.711680 22861 raft_consensus.cc:348] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.711798 22861 raft_consensus.cc:374] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:54.711838 22861 raft_consensus.cc:729] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ddf23f63a94d92a71bab927a4ef9a0, State: Initialized, Role: FOLLOWER
I20260619 06:48:54.711870 22863 consensus_queue.cc:260] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.711889 22862 raft_consensus.cc:348] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.711984 22862 raft_consensus.cc:374] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:54.711966 22861 consensus_queue.cc:260] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.712025 22862 raft_consensus.cc:729] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa5e57e1c354da39237cfb57987017d, State: Initialized, Role: FOLLOWER
I20260619 06:48:54.712163 22863 ts_tablet_manager.cc:1428] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:54.712152 22862 consensus_queue.cc:260] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.712222 22861 ts_tablet_manager.cc:1428] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:54.712356 22872 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.712423 22862 ts_tablet_manager.cc:1428] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:54.712388 22867 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.712479 22872 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:54.712503 22867 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:54.713387 22869 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.713505 22869 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:54.826658 22872 raft_consensus.cc:482] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:54.826782 22872 raft_consensus.cc:504] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.827236 22872 leader_election.cc:288] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901), bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975)
I20260619 06:48:54.827732 22647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347643bb76c04afc980a7628f2e9e066" candidate_uuid: "9222f1e019294a17b45f1ca7fea91cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" is_pre_election: true
I20260619 06:48:54.827732 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347643bb76c04afc980a7628f2e9e066" candidate_uuid: "9222f1e019294a17b45f1ca7fea91cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa5e57e1c354da39237cfb57987017d" is_pre_election: true
I20260619 06:48:54.827883 22787 raft_consensus.cc:2454] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9222f1e019294a17b45f1ca7fea91cb1 in term 0.
I20260619 06:48:54.827893 22647 raft_consensus.cc:2454] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9222f1e019294a17b45f1ca7fea91cb1 in term 0.
I20260619 06:48:54.828183 22438 leader_election.cc:302] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9222f1e019294a17b45f1ca7fea91cb1, a5ddf23f63a94d92a71bab927a4ef9a0; no voters:
I20260619 06:48:54.828408 22872 raft_consensus.cc:2781] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:54.828493 22872 raft_consensus.cc:482] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:54.828552 22872 raft_consensus.cc:3037] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:54.829639 22872 raft_consensus.cc:504] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.830001 22872 leader_election.cc:288] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [CANDIDATE]: Term 1 election: Requested vote from peers a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901), bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975)
I20260619 06:48:54.830310 22647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347643bb76c04afc980a7628f2e9e066" candidate_uuid: "9222f1e019294a17b45f1ca7fea91cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0"
I20260619 06:48:54.830356 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347643bb76c04afc980a7628f2e9e066" candidate_uuid: "9222f1e019294a17b45f1ca7fea91cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa5e57e1c354da39237cfb57987017d"
I20260619 06:48:54.830447 22647 raft_consensus.cc:3037] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:54.830482 22787 raft_consensus.cc:3037] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:54.831280 22647 raft_consensus.cc:2454] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9222f1e019294a17b45f1ca7fea91cb1 in term 1.
I20260619 06:48:54.831372 22787 raft_consensus.cc:2454] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9222f1e019294a17b45f1ca7fea91cb1 in term 1.
I20260619 06:48:54.831566 22438 leader_election.cc:302] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9222f1e019294a17b45f1ca7fea91cb1, a5ddf23f63a94d92a71bab927a4ef9a0; no voters:
I20260619 06:48:54.831761 22872 raft_consensus.cc:2781] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:54.831864 22872 raft_consensus.cc:686] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 LEADER]: Becoming Leader. State: Replica: 9222f1e019294a17b45f1ca7fea91cb1, State: Running, Role: LEADER
I20260619 06:48:54.831997 22872 consensus_queue.cc:237] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:54.832652 22868 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: TxnStatusTablet state changed. Reason: New leader 9222f1e019294a17b45f1ca7fea91cb1. Latest consensus state: current_term: 1 leader_uuid: "9222f1e019294a17b45f1ca7fea91cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.832758 22868 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:48:54.833021 22903 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:48:54.833184 22903 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:48:54.833635 22348 catalog_manager.cc:5495] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193). New cstate: current_term: 1 leader_uuid: "9222f1e019294a17b45f1ca7fea91cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } health_report { overall_health: UNKNOWN } } }
I20260619 06:48:54.867449 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.863967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.14.116.193:46355)
0619 06:48:54.864086 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.864090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.864186 (+ 96us) server_negotiation.cc:407] Connection header received
0619 06:48:54.864359 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.864362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.864403 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.864467 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:54.864713 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.865192 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.866384 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.866702 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.866779 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.867071 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:54.867132 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:54.867189 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.867299 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.867341 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:48:54.867509 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.863885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46355 (local address 127.0.0.1:46066)
0619 06:48:54.864114 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.864129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.864238 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.864507 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.864510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.864518 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:54.864625 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.864631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.865292 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.865295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.866283 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.866293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.866801 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.866805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.866904 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.866955 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:54.867212 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:54.867224 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.867294 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.867372 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 06:48:54.869935 22787 raft_consensus.cc:1261] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Refusing update from remote peer 9222f1e019294a17b45f1ca7fea91cb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:54.869982 22647 raft_consensus.cc:1261] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Refusing update from remote peer 9222f1e019294a17b45f1ca7fea91cb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:54.870355 22868 consensus_queue.cc:1035] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:54.870556 22872 consensus_queue.cc:1035] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:54.871738 22869 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: TxnStatusTablet state changed. Reason: New leader 9222f1e019294a17b45f1ca7fea91cb1. Latest consensus state: current_term: 1 leader_uuid: "9222f1e019294a17b45f1ca7fea91cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.871783 22867 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: TxnStatusTablet state changed. Reason: New leader 9222f1e019294a17b45f1ca7fea91cb1. Latest consensus state: current_term: 1 leader_uuid: "9222f1e019294a17b45f1ca7fea91cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.871855 22869 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:54.871884 22867 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:54.873103 22868 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9222f1e019294a17b45f1ca7fea91cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.873206 22868 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:48:54.874536 22869 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9222f1e019294a17b45f1ca7fea91cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.874629 22869 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P bfa5e57e1c354da39237cfb57987017d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:54.876794 22868 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9222f1e019294a17b45f1ca7fea91cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.876910 22868 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P 9222f1e019294a17b45f1ca7fea91cb1: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:48:54.878244 22867 tablet_replica.cc:439] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9222f1e019294a17b45f1ca7fea91cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } } }
I20260619 06:48:54.878353 22867 tablet_replica.cc:442] T 347643bb76c04afc980a7628f2e9e066 P a5ddf23f63a94d92a71bab927a4ef9a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:48:54.883769 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.877821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.14.116.195:40975)
0619 06:48:54.879909 (+ 2088us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.879914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.879927 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:48:54.879994 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.879997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.880052 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.880164 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:54.880383 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.881189 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.882190 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.882579 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.882676 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.882890 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:54.883404 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:54.883513 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.883555 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.883628 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1943,"thread_start_us":72,"threads_started":1}
I20260619 06:48:54.893720 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.890309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46355 (local address 127.14.116.195:34321)
0619 06:48:54.890703 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.890714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.890824 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.891069 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.891072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.891080 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:54.891208 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.891213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.891973 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.891977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.892578 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.892585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.893027 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.893031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.893125 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.893167 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:54.893443 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:54.893451 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.893507 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.893580 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":101,"threads_started":1}
I20260619 06:48:54.893708 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.890388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:34321 (local address 127.14.116.193:46355)
0619 06:48:54.890505 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.890510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.890797 (+ 287us) server_negotiation.cc:407] Connection header received
0619 06:48:54.890934 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.890937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.890981 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.891051 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:54.891308 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.891868 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.892691 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.892927 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.893027 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.893296 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:54.893359 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:54.893436 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.893526 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.893576 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:48:54.899888 22647 raft_consensus.cc:1261] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Refusing update from remote peer bfa5e57e1c354da39237cfb57987017d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:54.900398 22507 raft_consensus.cc:1261] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Refusing update from remote peer bfa5e57e1c354da39237cfb57987017d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:54.900748 22891 consensus_queue.cc:1035] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:54.900892 22869 consensus_queue.cc:1035] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:54.922302 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.918661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:40975 (local address 127.14.116.193:40901)
0619 06:48:54.918951 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:54.918964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:54.919045 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:54.919244 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:54.919246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:54.919251 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:54.919330 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.919334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.920325 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.920331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.921174 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:54.921183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.921719 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.921722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:54.921778 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.921816 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:54.922078 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:54.922083 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:48:54.922131 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 06:48:54.922189 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":99,"threads_started":1}
I20260619 06:48:54.922307 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:54.918705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:40901 (local address 127.14.116.195:40975)
0619 06:48:54.918832 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.918836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.919027 (+ 191us) server_negotiation.cc:407] Connection header received
0619 06:48:54.919111 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.919114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.919159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.919241 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:54.919428 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.920231 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.921278 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.921621 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.921722 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:54.921928 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:54.922002 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:54.922071 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:54.922139 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 06:48:54.922186 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:48:54.950563 22349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:54.950726 22349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:54.952937 22349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f: Sending DeleteTablet for 3 replicas of tablet c63aebd0f1864f9fb896db480ef64136
I20260619 06:48:54.953470 22487 tablet_service.cc:1514] Processing DeleteTablet for tablet c63aebd0f1864f9fb896db480ef64136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:54 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:46050
I20260619 06:48:54.953743 22627 tablet_service.cc:1514] Processing DeleteTablet for tablet c63aebd0f1864f9fb896db480ef64136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:54 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:49240
I20260619 06:48:54.953920 22919 tablet_replica.cc:330] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: stopping tablet replica
I20260619 06:48:54.954082 22919 raft_consensus.cc:2229] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:54.954154 22767 tablet_service.cc:1514] Processing DeleteTablet for tablet c63aebd0f1864f9fb896db480ef64136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:54 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:59002
I20260619 06:48:54.954317 22919 raft_consensus.cc:2258] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:54.955010 22919 ts_tablet_manager.cc:1905] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:54.958024 22919 ts_tablet_manager.cc:1918] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:54.958163 22919 log.cc:1196] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/c63aebd0f1864f9fb896db480ef64136
I20260619 06:48:54.958616 22919 ts_tablet_manager.cc:1939] T c63aebd0f1864f9fb896db480ef64136 P 9222f1e019294a17b45f1ca7fea91cb1: Deleting consensus metadata
I20260619 06:48:54.959331 22337 catalog_manager.cc:4841] TS 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355): tablet c63aebd0f1864f9fb896db480ef64136 (table test-table [id=7e36c568b1834e33b3772e0db53841db]) successfully deleted
I20260619 06:48:54.959749 22921 tablet_replica.cc:330] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: stopping tablet replica
I20260619 06:48:54.959916 22921 raft_consensus.cc:2229] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:54.959954 22920 tablet_replica.cc:330] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: stopping tablet replica
I20260619 06:48:54.960085 22920 raft_consensus.cc:2229] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:54.960219 22920 raft_consensus.cc:2258] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:54.960338 22921 raft_consensus.cc:2258] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:54.960903 22920 ts_tablet_manager.cc:1905] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:54.961023 22921 ts_tablet_manager.cc:1905] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:54.963578 22921 ts_tablet_manager.cc:1918] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:54.963716 22921 log.cc:1196] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/c63aebd0f1864f9fb896db480ef64136
I20260619 06:48:54.963745 22920 ts_tablet_manager.cc:1918] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:54.963828 22920 log.cc:1196] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/c63aebd0f1864f9fb896db480ef64136
I20260619 06:48:54.964094 22921 ts_tablet_manager.cc:1939] T c63aebd0f1864f9fb896db480ef64136 P bfa5e57e1c354da39237cfb57987017d: Deleting consensus metadata
I20260619 06:48:54.964135 22920 ts_tablet_manager.cc:1939] T c63aebd0f1864f9fb896db480ef64136 P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting consensus metadata
I20260619 06:48:54.964795 22335 catalog_manager.cc:4841] TS bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975): tablet c63aebd0f1864f9fb896db480ef64136 (table test-table [id=7e36c568b1834e33b3772e0db53841db]) successfully deleted
I20260619 06:48:54.964797 22334 catalog_manager.cc:4841] TS a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901): tablet c63aebd0f1864f9fb896db480ef64136 (table test-table [id=7e36c568b1834e33b3772e0db53841db]) successfully deleted
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:54 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:48:55.000336 22399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:54.988381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.14.116.254:35567)
0619 06:48:54.988523 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:54.988528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:54.989093 (+ 565us) server_negotiation.cc:407] Connection header received
0619 06:48:54.989285 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:54.989289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:54.989337 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:54.989389 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:54.990408 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.991234 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.992663 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:54.993005 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:54.993082 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.000203 (+ 7121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47888: BlockingRecv error: recv got EOF from 127.0.0.1:47888 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 06:48:55 dist-test-slave-9x3b krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM
I20260619 06:48:55.011873 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.002429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.14.116.254:35567)
0619 06:48:55.002551 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:55.002556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:55.002743 (+ 187us) server_negotiation.cc:407] Connection header received
0619 06:48:55.002846 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:55.002849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:55.002902 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:55.002994 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:55.003706 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.004513 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.005616 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.005938 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.006034 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.008661 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:55.008664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:55.008666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:55.008689 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:55.010155 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:55.010575 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:55.010579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:55.010581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:55.010665 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:55.010973 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:55.010976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:55.010978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:55.011149 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:55.011285 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:55.011554 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 06:48:55.011689 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:48:55.014385 22349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:48:55.014799 22349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:55.019420 22487 tablet_service.cc:1467] Processing CreateTablet for tablet 881a66c411b04da2a45767f4e81d35ec (DEFAULT_TABLE table=test-table [id=c4fa2ca6ac8b495a8d01eaf27cc4d53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:55.019505 22627 tablet_service.cc:1467] Processing CreateTablet for tablet 881a66c411b04da2a45767f4e81d35ec (DEFAULT_TABLE table=test-table [id=c4fa2ca6ac8b495a8d01eaf27cc4d53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:55.019767 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881a66c411b04da2a45767f4e81d35ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:55.019773 22627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881a66c411b04da2a45767f4e81d35ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:55.022135 22861 tablet_bootstrap.cc:492] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: Bootstrap starting.
I20260619 06:48:55.022343 22767 tablet_service.cc:1467] Processing CreateTablet for tablet 881a66c411b04da2a45767f4e81d35ec (DEFAULT_TABLE table=test-table [id=c4fa2ca6ac8b495a8d01eaf27cc4d53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:55.022619 22767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 881a66c411b04da2a45767f4e81d35ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:55.023000 22861 tablet_bootstrap.cc:654] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:55.023798 22863 tablet_bootstrap.cc:492] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: Bootstrap starting.
I20260619 06:48:55.024462 22861 tablet_bootstrap.cc:492] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: No bootstrap required, opened a new log
I20260619 06:48:55.024533 22861 ts_tablet_manager.cc:1397] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:55.024564 22863 tablet_bootstrap.cc:654] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:55.024830 22862 tablet_bootstrap.cc:492] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: Bootstrap starting.
I20260619 06:48:55.024935 22861 raft_consensus.cc:348] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.025039 22861 raft_consensus.cc:374] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:55.025069 22861 raft_consensus.cc:729] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ddf23f63a94d92a71bab927a4ef9a0, State: Initialized, Role: FOLLOWER
I20260619 06:48:55.025180 22861 consensus_queue.cc:260] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.025409 22861 ts_tablet_manager.cc:1428] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:55.025696 22862 tablet_bootstrap.cc:654] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:55.027351 22862 tablet_bootstrap.cc:492] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: No bootstrap required, opened a new log
I20260619 06:48:55.027455 22862 ts_tablet_manager.cc:1397] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 06:48:55.027683 22863 tablet_bootstrap.cc:492] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: No bootstrap required, opened a new log
I20260619 06:48:55.027772 22863 ts_tablet_manager.cc:1397] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:48:55.027943 22862 raft_consensus.cc:348] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.028057 22862 raft_consensus.cc:374] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:55.028096 22862 raft_consensus.cc:729] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa5e57e1c354da39237cfb57987017d, State: Initialized, Role: FOLLOWER
I20260619 06:48:55.028132 22863 raft_consensus.cc:348] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.028239 22863 raft_consensus.cc:374] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:55.028275 22863 raft_consensus.cc:729] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9222f1e019294a17b45f1ca7fea91cb1, State: Initialized, Role: FOLLOWER
I20260619 06:48:55.028227 22862 consensus_queue.cc:260] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.028388 22863 consensus_queue.cc:260] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.028542 22862 ts_tablet_manager.cc:1428] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:55.028640 22863 ts_tablet_manager.cc:1428] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:48:55.112668 22869 raft_consensus.cc:482] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:55.112812 22869 raft_consensus.cc:504] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.113265 22869 leader_election.cc:288] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355), a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901)
I20260619 06:48:55.113715 22647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881a66c411b04da2a45767f4e81d35ec" candidate_uuid: "bfa5e57e1c354da39237cfb57987017d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" is_pre_election: true
I20260619 06:48:55.113718 22507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881a66c411b04da2a45767f4e81d35ec" candidate_uuid: "bfa5e57e1c354da39237cfb57987017d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9222f1e019294a17b45f1ca7fea91cb1" is_pre_election: true
I20260619 06:48:55.113873 22647 raft_consensus.cc:2454] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa5e57e1c354da39237cfb57987017d in term 0.
I20260619 06:48:55.113878 22507 raft_consensus.cc:2454] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa5e57e1c354da39237cfb57987017d in term 0.
I20260619 06:48:55.114158 22718 leader_election.cc:302] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ddf23f63a94d92a71bab927a4ef9a0, bfa5e57e1c354da39237cfb57987017d; no voters:
I20260619 06:48:55.114342 22869 raft_consensus.cc:2781] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:55.114408 22869 raft_consensus.cc:482] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:55.114454 22869 raft_consensus.cc:3037] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:55.115396 22869 raft_consensus.cc:504] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.115767 22869 leader_election.cc:288] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [CANDIDATE]: Term 1 election: Requested vote from peers 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355), a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901)
I20260619 06:48:55.116125 22507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881a66c411b04da2a45767f4e81d35ec" candidate_uuid: "bfa5e57e1c354da39237cfb57987017d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9222f1e019294a17b45f1ca7fea91cb1"
I20260619 06:48:55.116125 22647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "881a66c411b04da2a45767f4e81d35ec" candidate_uuid: "bfa5e57e1c354da39237cfb57987017d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0"
I20260619 06:48:55.116245 22647 raft_consensus.cc:3037] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:55.116245 22507 raft_consensus.cc:3037] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:55.117350 22647 raft_consensus.cc:2454] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa5e57e1c354da39237cfb57987017d in term 1.
I20260619 06:48:55.117365 22507 raft_consensus.cc:2454] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa5e57e1c354da39237cfb57987017d in term 1.
I20260619 06:48:55.117628 22718 leader_election.cc:302] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ddf23f63a94d92a71bab927a4ef9a0, bfa5e57e1c354da39237cfb57987017d; no voters:
I20260619 06:48:55.117766 22869 raft_consensus.cc:2781] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:55.117854 22869 raft_consensus.cc:686] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 1 LEADER]: Becoming Leader. State: Replica: bfa5e57e1c354da39237cfb57987017d, State: Running, Role: LEADER
I20260619 06:48:55.117996 22869 consensus_queue.cc:237] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.119783 22349 catalog_manager.cc:5495] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d reported cstate change: term changed from 0 to 1, leader changed from <none> to bfa5e57e1c354da39237cfb57987017d (127.14.116.195). New cstate: current_term: 1 leader_uuid: "bfa5e57e1c354da39237cfb57987017d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } health_report { overall_health: HEALTHY } } }
I20260619 06:48:55.150130 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.147091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.14.116.195:40975)
0619 06:48:55.147213 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:55.147219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:55.147272 (+ 53us) server_negotiation.cc:407] Connection header received
0619 06:48:55.147417 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:55.147420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:55.147470 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:55.147546 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:55.147801 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.148301 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.149140 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.149364 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.149470 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.149715 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:55.149783 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:55.149856 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:55.149947 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 06:48:55.149992 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:48:55.151914 22647 raft_consensus.cc:1261] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Refusing update from remote peer bfa5e57e1c354da39237cfb57987017d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:55.151932 22507 raft_consensus.cc:1261] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Refusing update from remote peer bfa5e57e1c354da39237cfb57987017d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:55.152400 22891 consensus_queue.cc:1035] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:55.152598 22869 consensus_queue.cc:1035] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:55.159305 22348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:55.159463 22348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:55.161262 22348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f: Sending DeleteTablet for 3 replicas of tablet 881a66c411b04da2a45767f4e81d35ec
I20260619 06:48:55.161751 22487 tablet_service.cc:1514] Processing DeleteTablet for tablet 881a66c411b04da2a45767f4e81d35ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:55 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:46050
I20260619 06:48:55.161916 22919 tablet_replica.cc:330] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: stopping tablet replica
I20260619 06:48:55.161919 22627 tablet_service.cc:1514] Processing DeleteTablet for tablet 881a66c411b04da2a45767f4e81d35ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:55 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:49240
I20260619 06:48:55.162020 22919 raft_consensus.cc:2229] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:55.162108 22919 raft_consensus.cc:2258] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:55.162081 22767 tablet_service.cc:1514] Processing DeleteTablet for tablet 881a66c411b04da2a45767f4e81d35ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:55 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:59002
I20260619 06:48:55.162231 22920 tablet_replica.cc:330] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: stopping tablet replica
I20260619 06:48:55.162331 22920 raft_consensus.cc:2229] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:55.162431 22920 raft_consensus.cc:2258] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:55.162573 22921 tablet_replica.cc:330] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: stopping tablet replica
I20260619 06:48:55.162673 22921 raft_consensus.cc:2229] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:55.162830 22921 raft_consensus.cc:2258] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:55.162734 22348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:48:55.162989 22920 ts_tablet_manager.cc:1905] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 06:48:55.163141 22348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:48:55.163808 22921 ts_tablet_manager.cc:1905] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:55.165241 22920 ts_tablet_manager.cc:1918] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:55.165326 22920 log.cc:1196] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/881a66c411b04da2a45767f4e81d35ec
I20260619 06:48:55.165591 22920 ts_tablet_manager.cc:1939] T 881a66c411b04da2a45767f4e81d35ec P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting consensus metadata
I20260619 06:48:55.166208 22334 catalog_manager.cc:4841] TS a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901): tablet 881a66c411b04da2a45767f4e81d35ec (table test-table [id=c4fa2ca6ac8b495a8d01eaf27cc4d53a]) successfully deleted
I20260619 06:48:55.166396 22921 ts_tablet_manager.cc:1918] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:55.166465 22921 log.cc:1196] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/881a66c411b04da2a45767f4e81d35ec
I20260619 06:48:55.166767 22921 ts_tablet_manager.cc:1939] T 881a66c411b04da2a45767f4e81d35ec P bfa5e57e1c354da39237cfb57987017d: Deleting consensus metadata
I20260619 06:48:55.167338 22335 catalog_manager.cc:4841] TS bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975): tablet 881a66c411b04da2a45767f4e81d35ec (table test-table [id=c4fa2ca6ac8b495a8d01eaf27cc4d53a]) successfully deleted
I20260619 06:48:55.167759 22919 ts_tablet_manager.cc:1905] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:55.168002 22487 tablet_service.cc:1467] Processing CreateTablet for tablet 475756badcde475faa298078fd8f90d4 (DEFAULT_TABLE table=test-table [id=9fbd1e6ed1334a27895f9f59c4ddfedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:55.168257 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475756badcde475faa298078fd8f90d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:55.169734 22919 ts_tablet_manager.cc:1918] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:48:55.169796 22919 log.cc:1196] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/881a66c411b04da2a45767f4e81d35ec
I20260619 06:48:55.170022 22863 tablet_bootstrap.cc:492] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1: Bootstrap starting.
I20260619 06:48:55.170058 22919 ts_tablet_manager.cc:1939] T 881a66c411b04da2a45767f4e81d35ec P 9222f1e019294a17b45f1ca7fea91cb1: Deleting consensus metadata
I20260619 06:48:55.170594 22337 catalog_manager.cc:4841] TS 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355): tablet 881a66c411b04da2a45767f4e81d35ec (table test-table [id=c4fa2ca6ac8b495a8d01eaf27cc4d53a]) successfully deleted
I20260619 06:48:55.170689 22767 tablet_service.cc:1467] Processing CreateTablet for tablet 475756badcde475faa298078fd8f90d4 (DEFAULT_TABLE table=test-table [id=9fbd1e6ed1334a27895f9f59c4ddfedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:55.170778 22863 tablet_bootstrap.cc:654] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:55.170926 22767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475756badcde475faa298078fd8f90d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:55.171819 22863 tablet_bootstrap.cc:492] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1: No bootstrap required, opened a new log
I20260619 06:48:55.171818 22627 tablet_service.cc:1467] Processing CreateTablet for tablet 475756badcde475faa298078fd8f90d4 (DEFAULT_TABLE table=test-table [id=9fbd1e6ed1334a27895f9f59c4ddfedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:48:55.171914 22863 ts_tablet_manager.cc:1397] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:55.172071 22627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475756badcde475faa298078fd8f90d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:55.172304 22863 raft_consensus.cc:348] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.172412 22863 raft_consensus.cc:374] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:55.172444 22863 raft_consensus.cc:729] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9222f1e019294a17b45f1ca7fea91cb1, State: Initialized, Role: FOLLOWER
I20260619 06:48:55.172554 22863 consensus_queue.cc:260] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.172820 22863 ts_tablet_manager.cc:1428] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:55.173067 22862 tablet_bootstrap.cc:492] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: Bootstrap starting.
I20260619 06:48:55.173854 22862 tablet_bootstrap.cc:654] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:55.174358 22861 tablet_bootstrap.cc:492] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: Bootstrap starting.
I20260619 06:48:55.174815 22862 tablet_bootstrap.cc:492] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: No bootstrap required, opened a new log
I20260619 06:48:55.174868 22862 ts_tablet_manager.cc:1397] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:55.175026 22861 tablet_bootstrap.cc:654] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:55.175140 22862 raft_consensus.cc:348] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.175210 22862 raft_consensus.cc:374] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:55.175230 22862 raft_consensus.cc:729] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa5e57e1c354da39237cfb57987017d, State: Initialized, Role: FOLLOWER
I20260619 06:48:55.175354 22862 consensus_queue.cc:260] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.175665 22862 ts_tablet_manager.cc:1428] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:48:55.175966 22861 tablet_bootstrap.cc:492] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: No bootstrap required, opened a new log
I20260619 06:48:55.176018 22861 ts_tablet_manager.cc:1397] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:48:55.176311 22861 raft_consensus.cc:348] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.176424 22861 raft_consensus.cc:374] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:55.176472 22861 raft_consensus.cc:729] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ddf23f63a94d92a71bab927a4ef9a0, State: Initialized, Role: FOLLOWER
I20260619 06:48:55.176604 22861 consensus_queue.cc:260] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.176858 22861 ts_tablet_manager.cc:1428] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:48:55.253068 22867 raft_consensus.cc:482] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:48:55.253199 22867 raft_consensus.cc:504] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.254166 22867 leader_election.cc:288] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355), bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975)
I20260619 06:48:55.257869 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.254512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:57743 (local address 127.14.116.195:40975)
0619 06:48:55.254640 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:55.254645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:55.254749 (+ 104us) server_negotiation.cc:407] Connection header received
0619 06:48:55.254912 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:55.254915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:55.254962 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:55.255046 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:55.255313 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.255833 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.256914 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.257328 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.257456 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.257514 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:55.257685 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 06:48:55.257753 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:48:55.257900 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.254390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:40975 (local address 127.14.116.194:57743)
0619 06:48:55.254654 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:55.254667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:55.254793 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:55.255059 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:55.255062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:55.255078 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:55.255205 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:55.255210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.255957 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.255960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:55.256786 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:55.256794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.257448 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.257451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:55.257553 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.257576 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:48:55.257671 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:48:55.257740 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260619 06:48:55.258116 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.254540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:56285 (local address 127.14.116.193:46355)
0619 06:48:55.254686 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:55.254691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:55.255026 (+ 335us) server_negotiation.cc:407] Connection header received
0619 06:48:55.255185 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:55.255188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:55.255233 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:55.255313 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:48:55.255576 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.256101 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.257066 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.257522 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.257649 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.257699 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:55.257903 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 06:48:55.257986 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:48:55.258244 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.254390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46355 (local address 127.14.116.194:56285)
0619 06:48:55.254937 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:55.254948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:55.255071 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:55.255324 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:55.255326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:55.255340 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:48:55.255470 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:55.255475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.256205 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.256211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:55.256949 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:55.256957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.257679 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.257683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:55.257786 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.257804 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 06:48:55.257889 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:48:55.258065 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":265,"thread_start_us":89,"threads_started":1}
I20260619 06:48:55.258251 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475756badcde475faa298078fd8f90d4" candidate_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa5e57e1c354da39237cfb57987017d" is_pre_election: true
I20260619 06:48:55.258396 22787 raft_consensus.cc:2454] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ddf23f63a94d92a71bab927a4ef9a0 in term 0.
I20260619 06:48:55.258792 22507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475756badcde475faa298078fd8f90d4" candidate_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9222f1e019294a17b45f1ca7fea91cb1" is_pre_election: true
I20260619 06:48:55.258944 22507 raft_consensus.cc:2454] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ddf23f63a94d92a71bab927a4ef9a0 in term 0.
I20260619 06:48:55.258996 22579 leader_election.cc:302] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ddf23f63a94d92a71bab927a4ef9a0, bfa5e57e1c354da39237cfb57987017d; no voters:
I20260619 06:48:55.259246 22867 raft_consensus.cc:2781] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:48:55.259308 22867 raft_consensus.cc:482] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:48:55.259371 22867 raft_consensus.cc:3037] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:55.260471 22867 raft_consensus.cc:504] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.260818 22867 leader_election.cc:288] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [CANDIDATE]: Term 1 election: Requested vote from peers 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355), bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975)
I20260619 06:48:55.261099 22507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475756badcde475faa298078fd8f90d4" candidate_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9222f1e019294a17b45f1ca7fea91cb1"
I20260619 06:48:55.261162 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475756badcde475faa298078fd8f90d4" candidate_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa5e57e1c354da39237cfb57987017d"
I20260619 06:48:55.261205 22507 raft_consensus.cc:3037] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:55.261293 22787 raft_consensus.cc:3037] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:55.262084 22507 raft_consensus.cc:2454] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ddf23f63a94d92a71bab927a4ef9a0 in term 1.
I20260619 06:48:55.262190 22787 raft_consensus.cc:2454] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ddf23f63a94d92a71bab927a4ef9a0 in term 1.
I20260619 06:48:55.262341 22581 leader_election.cc:302] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9222f1e019294a17b45f1ca7fea91cb1, a5ddf23f63a94d92a71bab927a4ef9a0; no voters:
I20260619 06:48:55.262503 22867 raft_consensus.cc:2781] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:55.262658 22867 raft_consensus.cc:686] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 LEADER]: Becoming Leader. State: Replica: a5ddf23f63a94d92a71bab927a4ef9a0, State: Running, Role: LEADER
I20260619 06:48:55.262892 22867 consensus_queue.cc:237] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } }
I20260619 06:48:55.264977 22349 catalog_manager.cc:5495] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194). New cstate: current_term: 1 leader_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ddf23f63a94d92a71bab927a4ef9a0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 } health_report { overall_health: UNKNOWN } } }
I20260619 06:48:55.279006 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.275487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.14.116.194:39901)
0619 06:48:55.275753 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:55.275764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:55.275778 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:55.275849 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:55.275851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:55.275896 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:55.276015 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:55.276233 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.276767 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.277594 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.277837 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.277944 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.278202 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:55.278628 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:55.278751 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:55.278809 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 06:48:55.278857 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260619 06:48:55.288218 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.285015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:60513 (local address 127.14.116.193:46355)
0619 06:48:55.285147 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:55.285152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:55.285276 (+ 124us) server_negotiation.cc:407] Connection header received
0619 06:48:55.285447 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:55.285449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:55.285490 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:55.285563 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:55.285828 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.286291 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.287182 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.287400 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.287508 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.287816 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:55.287882 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:55.287959 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:55.288043 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 06:48:55.288087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:48:55.288260 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.284875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46355 (local address 127.14.116.194:60513)
0619 06:48:55.285195 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:55.285207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:55.285319 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:55.285587 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:55.285589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:55.285596 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:55.285711 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:55.285715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.286392 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.286396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:55.287054 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:55.287062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.287519 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.287522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:55.287634 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.287672 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:55.287962 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:55.287970 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:48:55.288035 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 06:48:55.288107 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":123,"threads_started":1}
I20260619 06:48:55.294428 22507 raft_consensus.cc:1261] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Refusing update from remote peer a5ddf23f63a94d92a71bab927a4ef9a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:55.294495 22787 raft_consensus.cc:1261] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Refusing update from remote peer a5ddf23f63a94d92a71bab927a4ef9a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:48:55.294884 22867 consensus_queue.cc:1035] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9222f1e019294a17b45f1ca7fea91cb1" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:55.295432 22947 consensus_queue.cc:1035] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa5e57e1c354da39237cfb57987017d" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:48:55.311313 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.308323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:52797 (local address 127.14.116.194:39901)
0619 06:48:55.308591 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:55.308598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:55.308612 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:55.308653 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:55.308656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:55.308705 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:55.308779 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:48:55.309047 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.309536 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.310281 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.310599 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.310701 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.310903 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:48:55.310988 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:48:55.311042 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:55.311139 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 06:48:55.311187 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260619 06:48:55.311316 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.308264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:39901 (local address 127.14.116.193:52797)
0619 06:48:55.308375 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:55.308389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:55.308463 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:55.308805 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:55.308808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:55.308816 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:48:55.308932 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:55.308941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.309619 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.309621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:55.310187 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:55.310194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:55.310703 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:55.310706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:55.310767 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:55.310791 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:48:55.311050 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:48:55.311056 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:48:55.311123 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 06:48:55.311198 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 06:48:55.332800 22348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:55.332963 22348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:48:55.334879 22348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72e82ffeaa3d42d18824511f9deacb3f: Sending DeleteTablet for 3 replicas of tablet 475756badcde475faa298078fd8f90d4
I20260619 06:48:55.335418 22627 tablet_service.cc:1514] Processing DeleteTablet for tablet 475756badcde475faa298078fd8f90d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:55 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:49240
I20260619 06:48:55.335404 22487 tablet_service.cc:1514] Processing DeleteTablet for tablet 475756badcde475faa298078fd8f90d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:55 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:46050
I20260619 06:48:55.335497 22767 tablet_service.cc:1514] Processing DeleteTablet for tablet 475756badcde475faa298078fd8f90d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:48:55 UTC) from {username='oryx', principal='oryx/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:59002
I20260619 06:48:55.335568 22920 tablet_replica.cc:330] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: stopping tablet replica
I20260619 06:48:55.335569 22919 tablet_replica.cc:330] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1: stopping tablet replica
I20260619 06:48:55.335687 22921 tablet_replica.cc:330] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: stopping tablet replica
I20260619 06:48:55.335696 22920 raft_consensus.cc:2229] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:48:55.335701 22919 raft_consensus.cc:2229] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:55.335799 22919 raft_consensus.cc:2258] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:55.335796 22921 raft_consensus.cc:2229] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:48:55.335901 22921 raft_consensus.cc:2258] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:55.335975 22920 raft_consensus.cc:2258] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:48:55.336648 22921 ts_tablet_manager.cc:1905] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:55.336696 22920 ts_tablet_manager.cc:1905] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:55.338789 22920 ts_tablet_manager.cc:1918] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:55.338879 22920 log.cc:1196] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/475756badcde475faa298078fd8f90d4
I20260619 06:48:55.339114 22921 ts_tablet_manager.cc:1918] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:48:55.339155 22920 ts_tablet_manager.cc:1939] T 475756badcde475faa298078fd8f90d4 P a5ddf23f63a94d92a71bab927a4ef9a0: Deleting consensus metadata
I20260619 06:48:55.339192 22921 log.cc:1196] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/475756badcde475faa298078fd8f90d4
I20260619 06:48:55.339443 22921 ts_tablet_manager.cc:1939] T 475756badcde475faa298078fd8f90d4 P bfa5e57e1c354da39237cfb57987017d: Deleting consensus metadata
I20260619 06:48:55.339871 22334 catalog_manager.cc:4841] TS a5ddf23f63a94d92a71bab927a4ef9a0 (127.14.116.194:39901): tablet 475756badcde475faa298078fd8f90d4 (table test-table [id=9fbd1e6ed1334a27895f9f59c4ddfedd]) successfully deleted
I20260619 06:48:55.340210 22919 ts_tablet_manager.cc:1905] T 475756badcde475faa298078fd8f90d4 P 9222f1e019294a17b45f1ca7fea91cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:48:55.340637 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 22422
I20260619 06:48:55.341758 22335 catalog_manager.cc:4841] TS bfa5e57e1c354da39237cfb57987017d (127.14.116.195:40975): tablet 475756badcde475faa298078fd8f90d4 (table test-table [id=9fbd1e6ed1334a27895f9f59c4ddfedd]) successfully deleted
W20260619 06:48:55.349570 22337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:46355 (error 108)
I20260619 06:48:55.349859 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 22562
I20260619 06:48:55.351588 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:55.350863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:46355 (local address 127.0.0.1:46072)
0619 06:48:55.351407 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:55.351500 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":63,"threads_started":1}
W20260619 06:48:55.351763 22337 catalog_manager.cc:4584] TS 9222f1e019294a17b45f1ca7fea91cb1 (127.14.116.193:46355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 475756badcde475faa298078fd8f90d4: Network error: Client connection negotiation failed: client connection to 127.14.116.193:46355: connect: Connection refused (error 111)
I20260619 06:48:55.355945 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 22702
I20260619 06:48:55.364042 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 22318
2026-06-19T06:48:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3165 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:55 dist-test-slave-9x3b krb5kdc[22962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:55 dist-test-slave-9x3b krb5kdc[22962](info): set up 2 sockets
Jun 19 06:48:55 dist-test-slave-9x3b krb5kdc[22962](info): commencing operation
krb5kdc: starting...
W20260619 06:48:56.404397 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:56 dist-test-slave-9x3b krb5kdc[22962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:56.515995 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:33247
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:46285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:33247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:56.620944 22975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:56.621229 22975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:56.621296 22975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:56.625204 22975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:56.625289 22975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:56.625327 22975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:56.625365 22975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:56.625404 22975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:56.630421 22975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:33247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:33247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:56.631395 22975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:56.632366 22975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:56.640769 22980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:56.640779 22981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:56.640854 22975 server_base.cc:1034] running on GCE node
W20260619 06:48:56.640779 22983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:56.641362 22975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:56.642357 22975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:56.643545 22975 hybrid_clock.cc:648] HybridClock initialized: now 1781851736643528 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T06:48:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1287 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:48:56 dist-test-slave-9x3b krb5kdc[22988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:48:56 dist-test-slave-9x3b krb5kdc[22988](info): set up 2 sockets
Jun 19 06:48:56 dist-test-slave-9x3b krb5kdc[22988](info): commencing operation
krb5kdc: starting...
W20260619 06:48:57.684437 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:48:57 dist-test-slave-9x3b krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:48:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:48:57.796211 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38717
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:38717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:57.899966 23001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:57.900241 23001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:57.900305 23001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:57.904245 23001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:57.904314 23001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:57.904352 23001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:57.904407 23001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:57.904443 23001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:57.909484 23001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:38717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:57.910408 23001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:57.911319 23001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:57.919232 23009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:57.919301 23006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:57.919232 23007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:57.919507 23001 server_base.cc:1034] running on GCE node
I20260619 06:48:57.920097 23001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:57.921100 23001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:57.922251 23001 hybrid_clock.cc:648] HybridClock initialized: now 1781851737922239 us; error 24 us; skew 500 ppm
Jun 19 06:48:57 dist-test-slave-9x3b krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851737, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:57.924894 23001 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:57.925626 23001 webserver.cc:460] Webserver started at http://127.14.116.254:33211/ using document root <none> and password file <none>
I20260619 06:48:57.926098 23001 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:57.926153 23001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:57.926301 23001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:57.928185 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "3fb022ae9c2a4626b51f03ae19d50e97"
format_stamp: "Formatted at 2026-06-19 06:48:57 on dist-test-slave-9x3b"
server_key: "f90ba26c8cc0deb0876816776e7567d3"
server_key_iv: "1a57ba6d26d5b23b58ec2dd5103996f2"
server_key_version: "encryptionkey@0"
I20260619 06:48:57.928656 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "3fb022ae9c2a4626b51f03ae19d50e97"
format_stamp: "Formatted at 2026-06-19 06:48:57 on dist-test-slave-9x3b"
server_key: "f90ba26c8cc0deb0876816776e7567d3"
server_key_iv: "1a57ba6d26d5b23b58ec2dd5103996f2"
server_key_version: "encryptionkey@0"
I20260619 06:48:57.932142 23001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:48:57.934320 23016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:57.935426 23001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:57.935544 23001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "3fb022ae9c2a4626b51f03ae19d50e97"
format_stamp: "Formatted at 2026-06-19 06:48:57 on dist-test-slave-9x3b"
server_key: "f90ba26c8cc0deb0876816776e7567d3"
server_key_iv: "1a57ba6d26d5b23b58ec2dd5103996f2"
server_key_version: "encryptionkey@0"
I20260619 06:48:57.935684 23001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:57.959471 23001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:57.962675 23001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:57.962882 23001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:57.969265 23001 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:38717
I20260619 06:48:57.969277 23068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:38717 every 8 connection(s)
I20260619 06:48:57.970144 23001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:57.971833 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 23001
I20260619 06:48:57.971977 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:48:57.972319 14803 external_mini_cluster.cc:1428] Setting key d3218846a6eaf49aad423c5d445f4df9
I20260619 06:48:57.973044 23069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:48:57.979131 23069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: Bootstrap starting.
Jun 19 06:48:57 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:57.981606 23069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: Neither blocks nor log segments found. Creating new log.
I20260619 06:48:57.982476 23069 log.cc:824] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:57.984211 23069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: No bootstrap required, opened a new log
I20260619 06:48:57.985795 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:57.973573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.14.116.254:38717)
0619 06:48:57.973947 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:57.973964 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:57.974006 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:48:57.974689 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:57.974706 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:57.975006 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:57.975280 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:57.976193 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:57.977128 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:57.978326 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:57.978757 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:57.978872 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:57.981475 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:57.981480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:57.981491 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:57.981515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:57.983155 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:57.983593 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:57.983599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:57.983603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:57.983757 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:57.984039 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:57.984042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:57.984043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:57.984444 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:57.984637 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:57.984897 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 06:48:57.985157 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":133,"threads_started":1}
I20260619 06:48:57.987914 23069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } }
I20260619 06:48:57.988153 23069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:48:57.988236 23069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb022ae9c2a4626b51f03ae19d50e97, State: Initialized, Role: FOLLOWER
I20260619 06:48:57.988606 23069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } }
I20260619 06:48:57.988740 23069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:57.988818 23069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:57.988910 23069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:48:57.989709 23069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } }
I20260619 06:48:57.990020 23069 leader_election.cc:302] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fb022ae9c2a4626b51f03ae19d50e97; no voters:
I20260619 06:48:57.990288 23069 leader_election.cc:288] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:48:57.990417 23074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:48:57.990658 23074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb022ae9c2a4626b51f03ae19d50e97, State: Running, Role: LEADER
I20260619 06:48:57.991005 23074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } }
I20260619 06:48:57.991549 23069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:57.992722 23075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } } }
I20260619 06:48:57.992712 23076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fb022ae9c2a4626b51f03ae19d50e97. Latest consensus state: current_term: 1 leader_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } } }
I20260619 06:48:57.992858 23075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:57.992858 23076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:57.993239 23083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:57.995728 23083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:58.002074 23083 catalog_manager.cc:1347] Generated new cluster ID: f1546cd973a544a1b2e7d635012123ab
I20260619 06:48:58.002148 23083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:58.010159 23083 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:48:58.011162 23083 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:58.015992 23083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: Generated new TSK 0
I20260619 06:48:58.016587 23083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:48:58.044138 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38717
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:58.150645 23095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:58.150869 23095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:58.150943 23095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:58.154877 23095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:58.154924 23095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:58.155023 23095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:58.160056 23095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:58.160982 23095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:58.161907 23095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:58.170848 23100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:58.170934 23095 server_base.cc:1034] running on GCE node
W20260619 06:48:58.170831 23103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:58.170831 23101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:58.171411 23095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:58.172030 23095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:58.173208 23095 hybrid_clock.cc:648] HybridClock initialized: now 1781851738173194 us; error 25 us; skew 500 ppm
Jun 19 06:48:58 dist-test-slave-9x3b krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:58.175806 23095 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:48:58.176571 23095 webserver.cc:460] Webserver started at http://127.14.116.193:37293/ using document root <none> and password file <none>
I20260619 06:48:58.177085 23095 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:58.177168 23095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:58.177386 23095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:58.179322 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "a670e59eda9241b5857420f865002881"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "8488db4e54d98457afeac63e15cd22be"
server_key_iv: "bb5029ac6d1bdbfa3494155072c6eb6a"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.179873 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "a670e59eda9241b5857420f865002881"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "8488db4e54d98457afeac63e15cd22be"
server_key_iv: "bb5029ac6d1bdbfa3494155072c6eb6a"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.183218 23095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:48:58.185505 23110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.186631 23095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:58.186751 23095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "a670e59eda9241b5857420f865002881"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "8488db4e54d98457afeac63e15cd22be"
server_key_iv: "bb5029ac6d1bdbfa3494155072c6eb6a"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.186851 23095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:58.202162 23095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:58.204744 23095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:58.204936 23095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:58.205475 23095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:58.206223 23095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:58.206295 23095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.206362 23095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:58.206408 23095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.214848 23095 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:46605
I20260619 06:48:58.214893 23223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:46605 every 8 connection(s)
I20260619 06:48:58.215716 23095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:58.219878 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 23095
I20260619 06:48:58.220013 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:48:58.220293 14803 external_mini_cluster.cc:1428] Setting key aea2f1647ef3ae7d85c0ec143fe70894
Jun 19 06:48:58 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
I20260619 06:48:58.229087 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:58.217365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:35045 (local address 127.14.116.254:38717)
0619 06:48:58.217513 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:58.217516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:58.218450 (+ 934us) server_negotiation.cc:407] Connection header received
0619 06:48:58.219228 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:58.219231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:58.219273 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:58.219343 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:58.220607 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.221252 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.222442 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.222866 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.222986 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:58.225612 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:58.225615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:58.225618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:58.225643 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:58.227092 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:58.227598 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:58.227603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:58.227605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:58.227705 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:58.228023 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:58.228027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:58.228029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:58.228211 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:58.228313 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:58.228784 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 06:48:58.228918 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:48:58.229884 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:58.217725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:35045)
0619 06:48:58.218319 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:58.218356 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:58.219062 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:58.219501 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:58.219507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:58.219895 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:58.220472 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:58.220487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.221407 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.221410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:58.222311 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:58.222320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.223035 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.223042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:58.223263 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:58.223900 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:58.223921 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:58.225456 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:58.227213 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:58.227218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:58.227231 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:58.227502 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:58.227798 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:58.227801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:58.227804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:58.227935 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:58.228310 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:58.228316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:58.228558 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 06:48:58.228736 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 06:48:58.229041 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":109,"threads_started":1}
I20260619 06:48:58.231036 23224 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38717
I20260619 06:48:58.231274 23224 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
I20260619 06:48:58.231837 23224 heartbeater.cc:511] Master 127.14.116.254:38717 requested a full tablet report, sending...
I20260619 06:48:58.233533 23033 ts_manager.cc:194] Registered new tserver with Master: a670e59eda9241b5857420f865002881 (127.14.116.193:46605)
I20260619 06:48:58.235203 23033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:35045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:48:58.246232 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38717
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:58.351665 23229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:58.351891 23229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:58.351935 23229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:58.355926 23229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:58.355973 23229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:58.356029 23229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:58.360986 23229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:58.361840 23229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:58.362747 23229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:58.371532 23234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:58.371532 23237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:58.371556 23235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:58.371658 23229 server_base.cc:1034] running on GCE node
I20260619 06:48:58.372196 23229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:58.372762 23229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:58.373912 23229 hybrid_clock.cc:648] HybridClock initialized: now 1781851738373882 us; error 43 us; skew 500 ppm
Jun 19 06:48:58 dist-test-slave-9x3b krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:58.376637 23229 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:48:58.377450 23229 webserver.cc:460] Webserver started at http://127.14.116.194:44375/ using document root <none> and password file <none>
I20260619 06:48:58.377970 23229 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:58.378058 23229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:58.378295 23229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:58.380229 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "6d40f2c5f1fc41b4a3745937b4b7593a"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "15bfbb8766820c85009a130aceff85d6"
server_key_iv: "c627ce1694cd74fb723ad7b81f4463a1"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.380759 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "6d40f2c5f1fc41b4a3745937b4b7593a"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "15bfbb8766820c85009a130aceff85d6"
server_key_iv: "c627ce1694cd74fb723ad7b81f4463a1"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.384065 23229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:48:58.386242 23244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.387310 23229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:58.387440 23229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "6d40f2c5f1fc41b4a3745937b4b7593a"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "15bfbb8766820c85009a130aceff85d6"
server_key_iv: "c627ce1694cd74fb723ad7b81f4463a1"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.387543 23229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:58.398311 23229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:58.401353 23229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:58.401535 23229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:58.402034 23229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:58.402760 23229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:58.402827 23229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.402889 23229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:58.402941 23229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.411339 23229 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:38211
I20260619 06:48:58.411366 23357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:38211 every 8 connection(s)
I20260619 06:48:58.412168 23229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
Jun 19 06:48:58 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:58.421890 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 23229
I20260619 06:48:58.422024 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:48:58.422349 14803 external_mini_cluster.cc:1428] Setting key 3f9591ad4ca826af2ab03920e4d5affc
I20260619 06:48:58.424314 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:58.413861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:43059 (local address 127.14.116.254:38717)
0619 06:48:58.413992 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:58.413995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:58.414928 (+ 933us) server_negotiation.cc:407] Connection header received
0619 06:48:58.415749 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:58.415752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:58.415795 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:58.415874 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:58.416914 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.417420 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.418449 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.418655 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.418727 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:58.420949 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:58.420952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:58.420954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:58.420975 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:58.422368 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:58.422843 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:58.422849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:58.422854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:58.422949 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:58.423275 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:58.423278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:58.423280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:58.423465 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:58.423564 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:58.424045 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 06:48:58.424156 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:48:58.425071 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:58.414210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:43059)
0619 06:48:58.414813 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:58.414850 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:58.415552 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:58.416025 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:58.416031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:58.416348 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:58.416793 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:58.416805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.417519 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.417522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:58.418350 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:58.418359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.418770 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.418772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:58.418899 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:58.419470 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:58.419485 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:58.420811 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:58.422492 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:58.422497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:58.422506 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:58.422745 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:58.423041 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:58.423045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:58.423048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:58.423183 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:58.423571 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:58.423576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:58.423829 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 06:48:58.424005 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 06:48:58.424286 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":105,"threads_started":1}
I20260619 06:48:58.426285 23358 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38717
I20260619 06:48:58.426501 23358 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:58.426915 23358 heartbeater.cc:511] Master 127.14.116.254:38717 requested a full tablet report, sending...
I20260619 06:48:58.427896 23033 ts_manager.cc:194] Registered new tserver with Master: 6d40f2c5f1fc41b4a3745937b4b7593a (127.14.116.194:38211)
I20260619 06:48:58.428452 23033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:43059
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:48:58.446069 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38717
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:58.551208 23363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:58.551427 23363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:58.551466 23363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:58.555369 23363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:58.555418 23363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:58.555516 23363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:58.560473 23363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:58.561379 23363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:58.562281 23363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:58.571225 23368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:58.571225 23369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:58.571259 23371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:58.571321 23363 server_base.cc:1034] running on GCE node
I20260619 06:48:58.571839 23363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:58.572407 23363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:58.573576 23363 hybrid_clock.cc:648] HybridClock initialized: now 1781851738573569 us; error 32 us; skew 500 ppm
Jun 19 06:48:58 dist-test-slave-9x3b krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:58.576181 23363 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:48:58.576932 23363 webserver.cc:460] Webserver started at http://127.14.116.195:33341/ using document root <none> and password file <none>
I20260619 06:48:58.577441 23363 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:58.577497 23363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:58.577749 23363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:48:58.579581 23363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "a48ca9b297764178907bf0e02cb3f1de"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "8616e88a72ab25da7e7aea2b20e9b54b"
server_key_iv: "8911e350a625f4dc4b7d506bedb07b42"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.580164 23363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "a48ca9b297764178907bf0e02cb3f1de"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "8616e88a72ab25da7e7aea2b20e9b54b"
server_key_iv: "8911e350a625f4dc4b7d506bedb07b42"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.583472 23363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:48:58.585669 23378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.586724 23363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:58.586833 23363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "a48ca9b297764178907bf0e02cb3f1de"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "8616e88a72ab25da7e7aea2b20e9b54b"
server_key_iv: "8911e350a625f4dc4b7d506bedb07b42"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.586927 23363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:58.604887 23363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:58.607441 23363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:58.607677 23363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:58.608390 23363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:58.609200 23363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:58.609244 23363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.609320 23363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:58.609360 23363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.617779 23363 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:39391
I20260619 06:48:58.617803 23491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:39391 every 8 connection(s)
I20260619 06:48:58.618604 23363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:58.621577 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 23363
I20260619 06:48:58.621697 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:48:58.621994 14803 external_mini_cluster.cc:1428] Setting key ac3cc2a058810ff05450c0010ac39f61
Jun 19 06:48:58 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:48:58.631186 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:58.620331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:59443 (local address 127.14.116.254:38717)
0619 06:48:58.620463 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:58.620467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:58.621451 (+ 984us) server_negotiation.cc:407] Connection header received
0619 06:48:58.622539 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:58.622542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:58.622587 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:58.622685 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:58.624113 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.624636 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.625677 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.625885 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.625963 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:58.628212 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:58.628215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:58.628217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:58.628238 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:58.629641 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:58.630035 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:58.630038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:58.630039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:58.630104 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:58.630365 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:58.630367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:58.630368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:58.630525 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:58.630602 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:58.630991 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 06:48:58.631077 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:48:58.631987 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:58.620681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:59443)
0619 06:48:58.621275 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:58.621339 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:58.622336 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:58.622851 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:58.622860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:58.623359 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:58.623966 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:58.623984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.624733 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.624735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:58.625574 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:58.625582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:58.626003 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:58.626005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:58.626135 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:58.626634 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:58.626648 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:58.627989 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:48:58.629730 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:58.629733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:58.629742 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:58.629958 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:58.630175 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:48:58.630177 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:48:58.630178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:48:58.630294 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:48:58.630612 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:48:58.630616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:48:58.630809 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 06:48:58.630950 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:48:58.631163 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":107,"threads_started":1}
I20260619 06:48:58.633013 23492 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38717
I20260619 06:48:58.633222 23492 heartbeater.cc:464] Registering TS with master...
I20260619 06:48:58.633644 23492 heartbeater.cc:511] Master 127.14.116.254:38717 requested a full tablet report, sending...
I20260619 06:48:58.634596 23033 ts_manager.cc:194] Registered new tserver with Master: a48ca9b297764178907bf0e02cb3f1de (127.14.116.195:39391)
I20260619 06:48:58.635223 23033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:59443
I20260619 06:48:58.635677 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab.
Entry for principal oryx/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab.
WARNING: no policy specified for oryx/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab.
Entry for principal oryx/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab.
WARNING: no policy specified for oryx/127.14.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab.
Entry for principal oryx/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab.
I20260619 06:48:58.700176 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 23095
I20260619 06:48:58.705426 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 23229
I20260619 06:48:58.710599 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 23363
I20260619 06:48:58.715704 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 23001
I20260619 06:48:58.721483 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38717
--webserver_interface=127.14.116.254
--webserver_port=33211
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:38717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:48:58.827809 23501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:58.828028 23501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:58.828102 23501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:58.832015 23501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:48:58.832062 23501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:58.832082 23501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:58.832130 23501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:48:58.832156 23501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:48:58.837224 23501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:38717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=33211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:58.838162 23501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:58.839080 23501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:58.847005 23509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:58.847043 23507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:58.847095 23506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:58.847141 23501 server_base.cc:1034] running on GCE node
I20260619 06:48:58.847831 23501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:58.848874 23501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:58.850026 23501 hybrid_clock.cc:648] HybridClock initialized: now 1781851738850008 us; error 32 us; skew 500 ppm
Jun 19 06:48:58 dist-test-slave-9x3b krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:58.852579 23501 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:48:58.853338 23501 webserver.cc:460] Webserver started at http://127.14.116.254:33211/ using document root <none> and password file <none>
I20260619 06:48:58.853842 23501 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:58.853930 23501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:58.857470 23501 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:58.859479 23516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:58.860708 23501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:58.860837 23501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "3fb022ae9c2a4626b51f03ae19d50e97"
format_stamp: "Formatted at 2026-06-19 06:48:57 on dist-test-slave-9x3b"
server_key: "f90ba26c8cc0deb0876816776e7567d3"
server_key_iv: "1a57ba6d26d5b23b58ec2dd5103996f2"
server_key_version: "encryptionkey@0"
I20260619 06:48:58.861325 23501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:58.870412 23501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:58.872990 23501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:58.873122 23501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:58.879242 23501 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:38717
I20260619 06:48:58.879294 23568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:38717 every 8 connection(s)
I20260619 06:48:58.880286 23501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:48:58.884231 23569 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:48:58.885725 23569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: Bootstrap starting.
I20260619 06:48:58.886461 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 23501
I20260619 06:48:58.886955 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:46605
--local_ip_for_outbound_sockets=127.14.116.193
--tserver_master_addrs=127.14.116.254:38717
--webserver_port=37293
--webserver_interface=127.14.116.193
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.116.193
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:48:58.902153 23569 log.cc:824] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: Log is configured to *not* fsync() on all Append() calls
I20260619 06:48:58.908198 23569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:48:58.908537 23569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: Bootstrap complete.
I20260619 06:48:58.912616 23569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } }
I20260619 06:48:58.913009 23569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb022ae9c2a4626b51f03ae19d50e97, State: Initialized, Role: FOLLOWER
I20260619 06:48:58.913496 23569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } }
I20260619 06:48:58.913635 23569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:48:58.913718 23569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:48:58.913831 23569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:48:58.915661 23569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } }
I20260619 06:48:58.915990 23569 leader_election.cc:302] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fb022ae9c2a4626b51f03ae19d50e97; no voters:
I20260619 06:48:58.916332 23569 leader_election.cc:288] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:48:58.916448 23573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:48:58.916678 23573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [term 2 LEADER]: Becoming Leader. State: Replica: 3fb022ae9c2a4626b51f03ae19d50e97, State: Running, Role: LEADER
I20260619 06:48:58.917037 23573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } }
I20260619 06:48:58.917420 23569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:48:58.918303 23574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } } }
I20260619 06:48:58.918414 23574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:58.919060 23575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fb022ae9c2a4626b51f03ae19d50e97. Latest consensus state: current_term: 2 leader_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fb022ae9c2a4626b51f03ae19d50e97" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38717 } } }
I20260619 06:48:58.919137 23575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97 [sys.catalog]: This master's current role is: LEADER
I20260619 06:48:58.922113 23590 catalog_manager.cc:1259] Loaded cluster ID: f1546cd973a544a1b2e7d635012123ab
I20260619 06:48:58.922168 23590 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: loading cluster ID for follower catalog manager: success
I20260619 06:48:58.924482 23590 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: acquiring CA information for follower catalog manager: success
I20260619 06:48:58.925769 23590 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:48:58.925899 23583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:48:58.926712 23583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:48:58.927122 23583 catalog_manager.cc:1259] Loaded cluster ID: f1546cd973a544a1b2e7d635012123ab
I20260619 06:48:58.927177 23583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:48:58.927799 23583 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:48:58.928229 23583 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3fb022ae9c2a4626b51f03ae19d50e97: Loaded TSK: 0
I20260619 06:48:58.928905 23583 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:48:58.993793 23571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:58.994041 23571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:58.994112 23571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:58.998051 23571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:58.998129 23571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:58.998214 23571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:48:59.003278 23571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.193.keytab
--principal=oryx/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:46605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=37293
--tserver_master_addrs=127.14.116.254:38717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:59.004238 23571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:59.005156 23571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:59.015065 23596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:59.015021 23597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:59.015373 23599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:59.015652 23571 server_base.cc:1034] running on GCE node
I20260619 06:48:59.016064 23571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:59.016798 23571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:59.018033 23571 hybrid_clock.cc:648] HybridClock initialized: now 1781851739018010 us; error 30 us; skew 500 ppm
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851739, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:59.021438 23571 init.cc:377] Logged in from keytab as oryx/127.14.116.193@KRBTEST.COM (short username oryx)
I20260619 06:48:59.022502 23571 webserver.cc:460] Webserver started at http://127.14.116.193:37293/ using document root <none> and password file <none>
I20260619 06:48:59.023205 23571 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:59.023312 23571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:59.028324 23571 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:48:59.030885 23606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:59.032151 23571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:48:59.032274 23571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "a670e59eda9241b5857420f865002881"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "8488db4e54d98457afeac63e15cd22be"
server_key_iv: "bb5029ac6d1bdbfa3494155072c6eb6a"
server_key_version: "encryptionkey@0"
I20260619 06:48:59.032799 23571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:59.046546 23571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:59.049691 23571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:59.049870 23571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:59.050381 23571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:59.051129 23571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:59.051204 23571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:59.051271 23571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:59.051322 23571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:59.060917 23571 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:46605
I20260619 06:48:59.062155 23571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:48:59.065497 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 23571
I20260619 06:48:59.065951 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:38211
--local_ip_for_outbound_sockets=127.14.116.194
--tserver_master_addrs=127.14.116.254:38717
--webserver_port=44375
--webserver_interface=127.14.116.194
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.116.194
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:48:59.067539 23719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:46605 every 8 connection(s)
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:48:59.084566 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.069605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:56081)
0619 06:48:59.070859 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:59.070915 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:59.072184 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:59.074009 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:59.074021 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:59.074645 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:59.075327 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.075351 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.077404 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.077410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.078941 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.078956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.079995 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.079999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.080183 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.080807 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:59.080839 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:59.083667 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":142,"threads_started":1}
W20260619 06:48:59.084833 23724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:59.071557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:56081 (local address 127.14.116.254:38717)
0619 06:48:59.072003 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.072023 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.072096 (+ 73us) server_negotiation.cc:407] Connection header received
0619 06:48:59.072656 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.072674 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.073093 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.073483 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.075553 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.077230 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.079075 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.079831 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.080013 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.083917 (+ 3904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:56081: BlockingRecv error: recv got EOF from 127.14.116.193:56081 (error 108)
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":118,"threads_started":1}
W20260619 06:48:59.085537 23720 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:38717 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.116.254:38717: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:48:59.092021 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.086015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:52885)
0619 06:48:59.086130 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:59.086145 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:59.086218 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:59.086574 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:59.086578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:59.086737 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:59.086860 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.086869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.087912 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.087916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.088807 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.088816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.089508 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.089512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.089617 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.090116 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:59.090130 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:59.091881 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:48:59.092106 23724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:59.086251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:52885 (local address 127.14.116.254:38717)
0619 06:48:59.086358 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.086362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.086375 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:48:59.086413 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.086416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.086461 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.086553 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.086985 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.087792 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.088913 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.089289 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.089393 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.091990 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:52885: BlockingRecv error: recv got EOF from 127.14.116.193:52885 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:48:59.097460 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.092765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:34947)
0619 06:48:59.092982 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:59.092993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:59.093085 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:59.093289 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:59.093292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:59.093444 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:59.093505 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.093510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.094167 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.094172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.094694 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.094701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.095134 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.095138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.095224 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.095709 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:59.095722 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:59.097332 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168}
W20260619 06:48:59.097560 23724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:59.092809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:34947 (local address 127.14.116.254:38717)
0619 06:48:59.092904 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.092908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.093052 (+ 144us) server_negotiation.cc:407] Connection header received
0619 06:48:59.093156 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.093159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.093200 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.093277 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.093605 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.094069 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.094805 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.095017 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.095124 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.097441 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:34947: BlockingRecv error: recv got EOF from 127.14.116.193:34947 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 06:48:59.097810 23720 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:48:59.180655 23722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:59.180888 23722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:59.180927 23722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:59.184824 23722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:59.184870 23722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:59.184930 23722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:48:59.189841 23722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.194.keytab
--principal=oryx/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:38211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=44375
--tserver_master_addrs=127.14.116.254:38717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:59.190702 23722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:59.191591 23722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:59.200716 23730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:59.200774 23729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:59.200753 23722 server_base.cc:1034] running on GCE node
W20260619 06:48:59.200716 23732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:59.201272 23722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:59.201840 23722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:59.202991 23722 hybrid_clock.cc:648] HybridClock initialized: now 1781851739202964 us; error 39 us; skew 500 ppm
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851739, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:59.205510 23722 init.cc:377] Logged in from keytab as oryx/127.14.116.194@KRBTEST.COM (short username oryx)
I20260619 06:48:59.206264 23722 webserver.cc:460] Webserver started at http://127.14.116.194:44375/ using document root <none> and password file <none>
I20260619 06:48:59.206776 23722 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:59.206880 23722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:59.210505 23722 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:59.212440 23739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:59.213553 23722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:48:59.213658 23722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "6d40f2c5f1fc41b4a3745937b4b7593a"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "15bfbb8766820c85009a130aceff85d6"
server_key_iv: "c627ce1694cd74fb723ad7b81f4463a1"
server_key_version: "encryptionkey@0"
I20260619 06:48:59.214133 23722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:59.236358 23722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:59.240197 23722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:59.240388 23722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:59.240976 23722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:59.241760 23722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:59.241802 23722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:59.241868 23722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:59.241899 23722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:59.250236 23722 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:38211
I20260619 06:48:59.250314 23852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:38211 every 8 connection(s)
I20260619 06:48:59.251169 23722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:48:59.254014 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 23722
I20260619 06:48:59.254452 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:39391
--local_ip_for_outbound_sockets=127.14.116.195
--tserver_master_addrs=127.14.116.254:38717
--webserver_port=33341
--webserver_interface=127.14.116.195
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.116.195
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:48:59.261950 23724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:59.252829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:46521 (local address 127.14.116.254:38717)
0619 06:48:59.252974 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.252978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.253921 (+ 943us) server_negotiation.cc:407] Connection header received
0619 06:48:59.254878 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.254884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.254952 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.255046 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.256180 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.257005 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.258096 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.258308 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.258419 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.261790 (+ 3371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:46521: BlockingRecv error: recv got EOF from 127.14.116.194:46521 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:48:59.262367 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.253174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:46521)
0619 06:48:59.253760 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:59.253831 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:59.254682 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:59.255188 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:59.255194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:59.255526 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:59.256032 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.256045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.257110 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.257112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.257957 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.257964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.258454 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.258458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.258624 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.259195 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:59.259209 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:59.261604 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":105,"threads_started":1}
W20260619 06:48:59.262856 23853 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:38717 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.116.254:38717: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:48:59.268538 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.263288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:45671)
0619 06:48:59.263390 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:59.263405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:59.263476 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:59.263787 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:59.263789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:59.263916 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:59.263996 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.264001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.265105 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.265107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.265708 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.265721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.266278 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.266281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.266347 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.266795 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:59.266806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:59.268460 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:48:59.268702 23724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:59.263381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:45671 (local address 127.14.116.254:38717)
0619 06:48:59.263523 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.263529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.263545 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:48:59.263588 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.263594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.263686 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.263778 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.264104 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.265001 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.265824 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.266166 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.266260 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.268597 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:45671: BlockingRecv error: recv got EOF from 127.14.116.194:45671 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:48:59.274379 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.269425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:47607)
0619 06:48:59.269725 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:59.269736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:59.269798 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:59.269995 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:59.269997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:59.270108 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:59.270166 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.270170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.271102 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.271104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.271591 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.271597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.272187 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.272189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.272243 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.272666 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:59.272675 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:59.274296 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238}
W20260619 06:48:59.274547 23853 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:48:59.274694 23724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:59.269587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:47607 (local address 127.14.116.254:38717)
0619 06:48:59.269697 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.269701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.269773 (+ 72us) server_negotiation.cc:407] Connection header received
0619 06:48:59.269869 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.269872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.269907 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.269976 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.270257 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.271007 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.271722 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.272086 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.272207 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.274427 (+ 2220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:47607: BlockingRecv error: recv got EOF from 127.14.116.194:47607 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 06:48:59.364790 23856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:48:59.365041 23856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:48:59.365108 23856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:48:59.369010 23856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:48:59.369061 23856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:48:59.369115 23856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:48:59.374068 23856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/krb5kdc/oryx_127.14.116.195.keytab
--principal=oryx/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:39391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=33341
--tserver_master_addrs=127.14.116.254:38717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:48:59.374995 23856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:48:59.375988 23856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:48:59.385167 23862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:59.385274 23856 server_base.cc:1034] running on GCE node
W20260619 06:48:59.385147 23861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:48:59.385147 23864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:48:59.385744 23856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:48:59.386333 23856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:48:59.387480 23856 hybrid_clock.cc:648] HybridClock initialized: now 1781851739387449 us; error 42 us; skew 500 ppm
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851739, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:48:59.390162 23856 init.cc:377] Logged in from keytab as oryx/127.14.116.195@KRBTEST.COM (short username oryx)
I20260619 06:48:59.390987 23856 webserver.cc:460] Webserver started at http://127.14.116.195:33341/ using document root <none> and password file <none>
I20260619 06:48:59.391516 23856 fs_manager.cc:359] Metadata directory not provided
I20260619 06:48:59.391605 23856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:48:59.395193 23856 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:48:59.397110 23871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:59.398224 23856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 06:48:59.398358 23856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "a48ca9b297764178907bf0e02cb3f1de"
format_stamp: "Formatted at 2026-06-19 06:48:58 on dist-test-slave-9x3b"
server_key: "8616e88a72ab25da7e7aea2b20e9b54b"
server_key_iv: "8911e350a625f4dc4b7d506bedb07b42"
server_key_version: "encryptionkey@0"
I20260619 06:48:59.398854 23856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:48:59.441766 23856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:48:59.445881 23856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:48:59.446075 23856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:48:59.446766 23856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:48:59.447700 23856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:48:59.447777 23856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:59.447841 23856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:48:59.447870 23856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:48:59.456666 23856 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:39391
I20260619 06:48:59.456691 23984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:39391 every 8 connection(s)
I20260619 06:48:59.457589 23856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:48:59.461083 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 23856
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:48:59.468201 23724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:59.459164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:56975 (local address 127.14.116.254:38717)
0619 06:48:59.459294 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.459298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.460391 (+ 1093us) server_negotiation.cc:407] Connection header received
0619 06:48:59.461168 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.461171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.461222 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.461288 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.462744 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.463255 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.464550 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.464798 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.464904 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.468083 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:56975: BlockingRecv error: recv got EOF from 127.14.116.195:56975 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
I20260619 06:48:59.468787 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.459559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:56975)
0619 06:48:59.460259 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:59.460298 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:59.460981 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:59.461465 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:59.461475 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:59.461968 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:59.462594 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.462609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.463372 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.463376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.464427 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.464436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.464935 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.464940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.465157 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.465707 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:59.465722 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:59.467989 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":106,"threads_started":1}
W20260619 06:48:59.469261 23985 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:38717 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.116.254:38717: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
I20260619 06:48:59.470338 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.461948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.14.116.254:38717)
0619 06:48:59.462190 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.462194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.462207 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:48:59.462244 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.462246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.462283 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.462378 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.463046 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.463557 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.464462 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.464739 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.464848 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.466200 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:48:59.466210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:48:59.466222 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:48:59.466245 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:48:59.467964 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:59.468468 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:59.468473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:59.468477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:59.468580 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:48:59.468858 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:48:59.468867 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:48:59.468868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:48:59.469312 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:48:59.469503 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 06:48:59.470064 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 06:48:59.470227 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:48:59.474902 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.469650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:35831)
0619 06:48:59.469750 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:59.469763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:59.469822 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:59.470146 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:59.470148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:59.470299 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:59.470415 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.470424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.471306 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.471309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.472134 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.472145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.472654 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.472658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.472745 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.473205 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:59.473221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:59.474778 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 06:48:59.474984 23724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:59.469702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:35831 (local address 127.14.116.254:38717)
0619 06:48:59.469807 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.469810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.469821 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:48:59.469907 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.469909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.469939 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.470030 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.470522 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.471190 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.472286 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.472535 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.472632 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.474880 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:35831: BlockingRecv error: recv got EOF from 127.14.116.195:35831 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:48:59 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:48:59.479489 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:48:59.475455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:33563)
0619 06:48:59.475554 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 06:48:59.475567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:48:59.475671 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:48:59.475873 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:48:59.475875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:48:59.475988 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:48:59.476047 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.476051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.476667 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.476669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.477150 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:48:59.477157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.477519 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.477522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:48:59.477572 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.477906 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:48:59.477916 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:48:59.479370 (+ 1454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:48:59.479570 23724 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:48:59.475505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:33563 (local address 127.14.116.254:38717)
0619 06:48:59.475645 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:48:59.475649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:48:59.475663 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:48:59.475759 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:48:59.475761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:48:59.475791 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:48:59.475851 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:48:59.476129 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.476586 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.477243 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:48:59.477436 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:48:59.477509 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:48:59.479466 (+ 1957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:33563: BlockingRecv error: recv got EOF from 127.14.116.195:33563 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 06:48:59.479746 23985 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 06:49:00 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:00 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:00.106644 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:00.098505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:55501)
0619 06:49:00.098801 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:00.098816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:00.098916 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:00.099144 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:00.099146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:00.099318 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:00.099425 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:00.099430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.100175 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.100177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:00.100732 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:00.100739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.101134 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.101136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:00.101204 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:00.104655 (+ 3451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:00.104667 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:00.106528 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
W20260619 06:49:00.106798 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:00.098586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:55501 (local address 127.14.116.254:38717)
0619 06:49:00.098839 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:00.098842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:00.098859 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:49:00.098997 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:00.098999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:00.099039 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:00.099106 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:00.099527 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.100088 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.100821 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.101044 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.101116 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:00.106665 (+ 5549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:55501: BlockingRecv error: recv got EOF from 127.14.116.193:55501 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":122,"threads_started":1}
Jun 19 06:49:00 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:00 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:00.280687 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:00.275161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:54323)
0619 06:49:00.275421 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:00.275435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:00.275528 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:00.275771 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:00.275773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:00.275957 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:00.276049 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:00.276054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.276979 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.276984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:00.277637 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:00.277645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.278094 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.278098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:00.278184 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:00.278649 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:00.278660 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:00.280511 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
W20260619 06:49:00.280747 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:00.275289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:54323 (local address 127.14.116.254:38717)
0619 06:49:00.275548 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:00.275551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:00.275565 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:49:00.275604 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:00.275625 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:00.275665 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:00.275733 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:00.276170 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.276872 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.277747 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.277987 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.278093 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:00.280638 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:54323: BlockingRecv error: recv got EOF from 127.14.116.194:54323 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Jun 19 06:49:00 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:00 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:00.485875 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:00.480478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:33139)
0619 06:49:00.480754 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:00.480767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:00.480880 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:00.481114 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:00.481116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:00.481286 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:00.481378 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:00.481382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.482081 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.482083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:00.482626 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:00.482632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.483020 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.483022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:00.483082 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:00.483556 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:00.483567 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:00.485755 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
W20260619 06:49:00.486114 23991 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:00.480600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:33139 (local address 127.14.116.254:38717)
0619 06:49:00.480740 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:00.480745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:00.480886 (+ 141us) server_negotiation.cc:407] Connection header received
0619 06:49:00.480962 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:00.480964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:00.481012 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:00.481075 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:00.481475 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.481994 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.482719 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:00.482930 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:00.483002 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:00.485884 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:33139: BlockingRecv error: recv got EOF from 127.14.116.195:33139 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:49:01 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:01 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:01.113287 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:01.107470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:48685)
0619 06:49:01.107788 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:01.107805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:01.107947 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:01.108224 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:01.108227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:01.108436 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:01.108527 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:01.108533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.109308 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.109312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:01.109880 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:01.109889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.110378 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.110382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:01.110469 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:01.111010 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:01.111022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:01.113081 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
W20260619 06:49:01.113328 23995 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:01.107518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:48685 (local address 127.14.116.254:38717)
0619 06:49:01.107785 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:01.107790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:01.107888 (+ 98us) server_negotiation.cc:407] Connection header received
0619 06:49:01.108053 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:01.108056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:01.108110 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:01.108203 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:01.108643 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.109185 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.110008 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.110265 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.110373 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:01.113208 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:48685: BlockingRecv error: recv got EOF from 127.14.116.193:48685 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":57,"threads_started":1}
Jun 19 06:49:01 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:01 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:01.287094 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:01.281560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:44055)
0619 06:49:01.281899 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:01.281912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:01.282056 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:01.282344 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:01.282347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:01.282543 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:01.282633 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:01.282638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.283394 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.283399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:01.283995 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:01.284003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.284454 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.284458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:01.284544 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:01.285037 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:01.285048 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:01.286907 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
W20260619 06:49:01.287151 23995 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:01.281596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:44055 (local address 127.14.116.254:38717)
0619 06:49:01.281748 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:01.281753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:01.282011 (+ 258us) server_negotiation.cc:407] Connection header received
0619 06:49:01.282159 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:01.282163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:01.282215 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:01.282299 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:01.282752 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.283281 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.284116 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.284343 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.284453 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:01.287042 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:44055: BlockingRecv error: recv got EOF from 127.14.116.194:44055 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 06:49:01 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:01 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:01.492005 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:01.486603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:38759)
0619 06:49:01.486878 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:01.486894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:01.487006 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:01.487243 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:01.487245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:01.487416 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:01.487511 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:01.487516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.488268 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.488270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:01.488815 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:01.488822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.489226 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.489228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:01.489285 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:01.489752 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:01.489763 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:01.491800 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
W20260619 06:49:01.492015 23995 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:01.486780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:38759 (local address 127.14.116.254:38717)
0619 06:49:01.486905 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:01.486909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:01.486958 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:49:01.487085 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:01.487088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:01.487130 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:01.487209 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:01.487603 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.488177 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.488921 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:01.489133 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:01.489209 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:01.491918 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:38759: BlockingRecv error: recv got EOF from 127.14.116.195:38759 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 06:49:02 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:02 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:02.119791 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:02.114017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:33853)
0619 06:49:02.114378 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:02.114391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:02.114488 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:02.114731 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:02.114733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:02.114907 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:02.115000 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:02.115005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.115807 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.115810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:02.116386 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:02.116395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.116825 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.116828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:02.116891 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:02.117397 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:02.117408 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:02.119586 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
W20260619 06:49:02.119865 23999 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:02.114146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:33853 (local address 127.14.116.254:38717)
0619 06:49:02.114386 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:02.114389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:02.114459 (+ 70us) server_negotiation.cc:407] Connection header received
0619 06:49:02.114570 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:02.114572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:02.114617 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:02.114695 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:02.115126 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.115690 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.116509 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.116734 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.116819 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:02.119732 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:33853: BlockingRecv error: recv got EOF from 127.14.116.193:33853 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
Jun 19 06:49:02 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:02 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:02.293247 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:02.287946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:52909)
0619 06:49:02.288210 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:02.288223 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:02.288324 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:02.288545 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:02.288547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:02.288721 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:02.288835 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:02.288840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.289559 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.289562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:02.290114 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:02.290120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.290549 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.290552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:02.290615 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:02.291082 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:02.291093 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:02.293082 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
W20260619 06:49:02.293336 23999 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:02.288066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:52909 (local address 127.14.116.254:38717)
0619 06:49:02.288221 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:02.288224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:02.288297 (+ 73us) server_negotiation.cc:407] Connection header received
0619 06:49:02.288403 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:02.288405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:02.288447 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:02.288512 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:02.288925 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.289443 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.290209 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.290444 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.290516 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:02.293209 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:52909: BlockingRecv error: recv got EOF from 127.14.116.194:52909 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 06:49:02 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:02 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:02.497721 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:02.492901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:59315)
0619 06:49:02.493168 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:02.493181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:02.493280 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:02.493488 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:02.493490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:02.493658 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:02.493747 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:02.493753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.494440 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.494442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:02.494982 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:02.494989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.495370 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.495372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:02.495434 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:02.495872 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:02.495883 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:02.497613 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
W20260619 06:49:02.497890 23999 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:02.492970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:59315 (local address 127.14.116.254:38717)
0619 06:49:02.493106 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:02.493110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:02.493245 (+ 135us) server_negotiation.cc:407] Connection header received
0619 06:49:02.493356 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:02.493358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:02.493402 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:02.493476 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:02.493836 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.494349 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.495078 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:02.495283 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:02.495354 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:02.497737 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:59315: BlockingRecv error: recv got EOF from 127.14.116.195:59315 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:49:03 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:03 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:03.126065 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:03.120552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:38901)
0619 06:49:03.120870 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:03.120884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:03.120983 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:03.121220 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:03.121222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:03.121400 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:03.121500 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:03.121505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.122505 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.122512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:03.123180 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:03.123190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.123594 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.123596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:03.123709 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:03.124179 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:03.124190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:03.125954 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
W20260619 06:49:03.126214 24003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:03.120656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:38901 (local address 127.14.116.254:38717)
0619 06:49:03.120893 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:03.120896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:03.120944 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:49:03.121066 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:03.121068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:03.121110 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:03.121186 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:03.121597 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.122387 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.123282 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.123502 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.123578 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:03.126083 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:38901: BlockingRecv error: recv got EOF from 127.14.116.193:38901 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
Jun 19 06:49:03 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:03 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:03.299170 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:03.293943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:42551)
0619 06:49:03.294189 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:03.294203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:03.294309 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:03.294528 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:03.294530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:03.294703 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:03.294793 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:03.294798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.295556 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.295558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:03.296117 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:03.296124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.296548 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.296550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:03.296621 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:03.297164 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:03.297176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:03.299004 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":66,"threads_started":1}
W20260619 06:49:03.299234 24003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:03.294022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:42551 (local address 127.14.116.254:38717)
0619 06:49:03.294195 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:03.294198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:03.294300 (+ 102us) server_negotiation.cc:407] Connection header received
0619 06:49:03.294385 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:03.294387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:03.294430 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:03.294497 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:03.294916 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.295460 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.296208 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.296448 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.296537 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:03.299120 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:42551: BlockingRecv error: recv got EOF from 127.14.116.194:42551 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 06:49:03 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:03 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:03.503484 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:03.498470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:43675)
0619 06:49:03.498723 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:03.498737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:03.498837 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:03.499015 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:03.499017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:03.499180 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:03.499266 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:03.499270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.500022 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.500024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:03.500560 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:03.500567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.500973 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.500975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:03.501031 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:03.501535 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:03.501547 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:03.503366 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":104,"threads_started":1}
W20260619 06:49:03.503600 24003 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:03.498516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:43675 (local address 127.14.116.254:38717)
0619 06:49:03.498640 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:03.498643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:03.498806 (+ 163us) server_negotiation.cc:407] Connection header received
0619 06:49:03.498897 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:03.498899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:03.498941 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:03.499003 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:03.499345 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.499931 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.500654 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:03.500883 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:03.500959 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:03.503489 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:43675: BlockingRecv error: recv got EOF from 127.14.116.195:43675 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 06:49:04 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:04 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:04.131920 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:04.126759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:58623)
0619 06:49:04.127018 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:04.127031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:04.127109 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:04.127338 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:04.127340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:04.127512 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:04.127657 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:04.127667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.128380 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.128382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:04.128945 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:04.128952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.129365 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.129367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:04.129423 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:04.129898 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:04.129909 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:04.131761 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
W20260619 06:49:04.132005 24007 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:04.126840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:58623 (local address 127.14.116.254:38717)
0619 06:49:04.127057 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:04.127060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:04.127072 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:49:04.127190 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:04.127192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:04.127231 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:04.127308 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:04.127761 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.128293 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.129047 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.129267 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.129350 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:04.131908 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:58623: BlockingRecv error: recv got EOF from 127.14.116.193:58623 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
Jun 19 06:49:04 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:04 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:04.305433 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:04.300073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:47969)
0619 06:49:04.300354 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:04.300368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:04.300647 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:04.300874 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:04.300876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:04.301057 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:04.301144 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:04.301149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.301868 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.301870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:04.302553 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:04.302567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.302970 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.302973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:04.303033 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:04.303512 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:04.303522 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:04.305325 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":123,"threads_started":1}
W20260619 06:49:04.305567 24007 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:04.300224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:47969 (local address 127.14.116.254:38717)
0619 06:49:04.300379 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:04.300383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:04.300597 (+ 214us) server_negotiation.cc:407] Connection header received
0619 06:49:04.300724 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:04.300726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:04.300767 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:04.300838 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:04.301234 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.301782 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.302655 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.302873 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.302952 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:04.305450 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:47969: BlockingRecv error: recv got EOF from 127.14.116.194:47969 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 06:49:04 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:04 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:04.509284 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:04.504298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:38937)
0619 06:49:04.504540 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:04.504553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:04.504653 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:04.504868 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:04.504870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:04.505053 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:04.505143 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:04.505147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.505855 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.505857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:04.506403 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:04.506411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.506898 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.506900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:04.506967 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:04.507442 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:04.507453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:04.509175 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
W20260619 06:49:04.509456 24007 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:04.504351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:38937 (local address 127.14.116.254:38717)
0619 06:49:04.504468 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:04.504473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:04.504628 (+ 155us) server_negotiation.cc:407] Connection header received
0619 06:49:04.504724 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:04.504726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:04.504770 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:04.504834 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:04.505235 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.505763 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.506592 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:04.506807 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:04.506884 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:04.509328 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:38937: BlockingRecv error: recv got EOF from 127.14.116.195:38937 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 06:49:05 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:05 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:05.141029 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:05.132675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:33481)
0619 06:49:05.132926 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:05.132942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:05.133027 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:05.133259 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:05.133261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:05.133436 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:05.133528 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:05.133533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.134239 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.134241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:05.134779 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:05.134786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.135190 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.135193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:05.135251 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:05.138829 (+ 3578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:05.138841 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:05.140913 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
W20260619 06:49:05.141187 24011 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:05.132751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:33481 (local address 127.14.116.254:38717)
0619 06:49:05.132974 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:05.132977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:05.132995 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:49:05.133109 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:05.133111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:05.133153 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:05.133228 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:05.133620 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.134142 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.134878 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.135095 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.135171 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:05.141055 (+ 5884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:33481: BlockingRecv error: recv got EOF from 127.14.116.193:33481 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
Jun 19 06:49:05 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:05 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:05.311296 24012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:05.306226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:56801)
0619 06:49:05.306506 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:05.306519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:05.306614 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:05.306819 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:05.306821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:05.306996 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:05.307082 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:05.307087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.307874 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.307876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:05.308437 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:05.308445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.308841 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.308843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:05.308900 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:05.309360 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:05.309371 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:05.311143 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":110,"threads_started":1}
W20260619 06:49:05.311443 24011 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:05.306271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:56801 (local address 127.14.116.254:38717)
0619 06:49:05.306421 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:05.306425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:05.306590 (+ 165us) server_negotiation.cc:407] Connection header received
0619 06:49:05.306678 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:05.306680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:05.306723 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:05.306786 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:05.307173 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.307781 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.308535 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.308748 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.308828 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:05.311315 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:56801: BlockingRecv error: recv got EOF from 127.14.116.194:56801 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 06:49:05 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:05 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:05.515307 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:05.510059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:55735)
0619 06:49:05.510321 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:05.510333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:05.510466 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:05.510738 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:05.510740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:05.510945 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:05.511032 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:05.511037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.511845 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.511848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:05.512371 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:05.512379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.512830 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.512833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:05.512916 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:05.513407 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:05.513418 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:05.515144 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
W20260619 06:49:05.515350 24011 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:05.510109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:55735 (local address 127.14.116.254:38717)
0619 06:49:05.510266 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:05.510270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:05.510404 (+ 134us) server_negotiation.cc:407] Connection header received
0619 06:49:05.510574 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:05.510577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:05.510628 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:05.510715 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:05.511150 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.511737 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.512505 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:05.512724 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:05.512829 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:05.515251 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:55735: BlockingRecv error: recv got EOF from 127.14.116.195:55735 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 06:49:06 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:06 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:06.147456 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:06.141892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:58117)
0619 06:49:06.142259 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:06.142274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:06.142375 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:06.142604 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:06.142606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:06.142785 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:06.142880 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:06.142885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.143633 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.143637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:06.144180 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:06.144189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.144584 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.144587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:06.144646 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:06.145236 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:06.145254 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:06.147346 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":80,"threads_started":1}
W20260619 06:49:06.147676 24015 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:06.141962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:58117 (local address 127.14.116.254:38717)
0619 06:49:06.142247 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:06.142250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:06.142342 (+ 92us) server_negotiation.cc:407] Connection header received
0619 06:49:06.142450 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:06.142453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:06.142494 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:06.142569 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:06.142974 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.143499 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.144272 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.144490 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.144563 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:06.147478 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:58117: BlockingRecv error: recv got EOF from 127.14.116.193:58117 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":123,"threads_started":1}
Jun 19 06:49:06 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:06 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:06.318008 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:06.312131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:34449)
0619 06:49:06.312550 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:06.312564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:06.312669 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:06.312900 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:06.312902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:06.313079 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:06.313172 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:06.313176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.313907 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.313910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:06.314496 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:06.314512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.314930 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.314932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:06.314995 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:06.315565 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:06.315577 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:06.317816 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":153,"threads_started":1}
W20260619 06:49:06.318105 24015 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:06.312258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:34449 (local address 127.14.116.254:38717)
0619 06:49:06.312398 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:06.312401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:06.312638 (+ 237us) server_negotiation.cc:407] Connection header received
0619 06:49:06.312750 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:06.312752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:06.312794 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:06.312865 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:06.313269 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.313776 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.314619 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.314834 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.314909 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:06.317975 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:34449: BlockingRecv error: recv got EOF from 127.14.116.194:34449 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 06:49:06 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:06 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:06.521870 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:06.516054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:48297)
0619 06:49:06.516347 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:06.516362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:06.516469 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:06.516691 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:06.516693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:06.516876 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:06.517035 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:06.517042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.517909 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.517911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:06.518546 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:06.518554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.519074 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.519077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:06.519137 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:06.519813 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:06.519824 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:06.521676 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
W20260619 06:49:06.521984 24015 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:06.516240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:48297 (local address 127.14.116.254:38717)
0619 06:49:06.516370 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:06.516373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:06.516445 (+ 72us) server_negotiation.cc:407] Connection header received
0619 06:49:06.516546 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:06.516548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:06.516590 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:06.516659 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:06.517155 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.517820 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.518657 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:06.518979 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:06.519063 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:06.521843 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:48297: BlockingRecv error: recv got EOF from 127.14.116.195:48297 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 06:49:07 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:07 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:07.153538 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:07.148204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:50375)
0619 06:49:07.148514 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:07.148527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:07.148615 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:07.148848 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:07.148850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:07.149020 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:07.149112 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:07.149117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.149807 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.149809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:07.150357 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:07.150364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.150838 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.150841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:07.150935 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:07.151520 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:07.151535 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:07.153426 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
W20260619 06:49:07.153683 24019 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:07.148401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:50375 (local address 127.14.116.254:38717)
0619 06:49:07.148649 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:07.148653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:07.148665 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:49:07.148698 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:07.148700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:07.148738 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:07.148810 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:07.149203 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.149720 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.150451 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.150713 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.150817 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:07.153551 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:50375: BlockingRecv error: recv got EOF from 127.14.116.193:50375 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":116,"threads_started":1}
Jun 19 06:49:07 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:07 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:07.324160 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:07.318793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:48861)
0619 06:49:07.319071 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:07.319088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:07.319232 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:07.319512 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:07.319514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:07.319706 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:07.319798 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:07.319804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.320651 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.320654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:07.321203 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:07.321209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.321656 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.321659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:07.321717 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:07.322213 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:07.322224 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:07.324006 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
W20260619 06:49:07.324275 24019 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:07.318891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:48861 (local address 127.14.116.254:38717)
0619 06:49:07.319050 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:07.319055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:07.319176 (+ 121us) server_negotiation.cc:407] Connection header received
0619 06:49:07.319326 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:07.319329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:07.319400 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:07.319480 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:07.319904 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.320563 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.321339 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.321558 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.321645 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:07.324129 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:48861: BlockingRecv error: recv got EOF from 127.14.116.194:48861 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 06:49:07 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:07 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:07.527534 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:07.522597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:45453)
0619 06:49:07.522838 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:07.522852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:07.522961 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:07.523162 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:07.523164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:07.523338 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:07.523426 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:07.523431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.524128 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.524130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:07.524664 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:07.524671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.525060 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.525063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:07.525122 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:07.525587 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:07.525598 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:07.527369 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
W20260619 06:49:07.527577 24019 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:07.522690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:45453 (local address 127.14.116.254:38717)
0619 06:49:07.522825 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:07.522829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:07.522910 (+ 81us) server_negotiation.cc:407] Connection header received
0619 06:49:07.523023 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:07.523025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:07.523067 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:07.523133 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:07.523524 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.524047 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.524762 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:07.524969 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:07.525044 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:07.527479 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:45453: BlockingRecv error: recv got EOF from 127.14.116.195:45453 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 06:49:08 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:08 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:08.159730 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:08.154441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:41835)
0619 06:49:08.154738 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:08.154760 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:08.154886 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:08.155181 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:08.155183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:08.155381 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:08.155474 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:08.155480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.156213 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.156215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:08.156783 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:08.156791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.157184 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.157186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:08.157245 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:08.157742 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:08.157753 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:08.159559 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
W20260619 06:49:08.159996 24023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:08.154484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:41835 (local address 127.14.116.254:38717)
0619 06:49:08.154811 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:08.154815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:08.154829 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:49:08.155001 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:08.155004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:08.155060 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:08.155150 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:08.155574 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.156111 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.156879 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.157094 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.157169 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:08.159746 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:41835: BlockingRecv error: recv got EOF from 127.14.116.193:41835 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":113,"threads_started":1}
Jun 19 06:49:08 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:08 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:08.330173 24024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:08.325006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:55877)
0619 06:49:08.325312 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:08.325338 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:08.325486 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:08.325780 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:08.325785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:08.326024 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:08.326145 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:08.326153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.326860 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.326862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:08.327407 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:08.327413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.327807 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.327810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:08.327870 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:08.328308 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:08.328318 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:08.330044 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":114,"threads_started":1}
W20260619 06:49:08.330255 24023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:08.325057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:55877 (local address 127.14.116.254:38717)
0619 06:49:08.325199 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:08.325204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:08.325454 (+ 250us) server_negotiation.cc:407] Connection header received
0619 06:49:08.325594 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:08.325597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:08.325654 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:08.325744 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:08.326270 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.326774 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.327498 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.327723 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.327794 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:08.330169 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:55877: BlockingRecv error: recv got EOF from 127.14.116.194:55877 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 06:49:08 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:08 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:08.533958 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:08.528398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:43677)
0619 06:49:08.528732 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:08.528745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:08.528867 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:08.529174 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:08.529177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:08.529376 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:08.529459 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:08.529465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.530195 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.530202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:08.530756 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:08.530763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.531212 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.531216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:08.531306 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:08.531828 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:08.531839 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:08.533740 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
W20260619 06:49:08.533957 24023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:08.528463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:43677 (local address 127.14.116.254:38717)
0619 06:49:08.528581 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:08.528585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:08.528824 (+ 239us) server_negotiation.cc:407] Connection header received
0619 06:49:08.528957 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:08.528962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:08.529014 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:08.529117 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:08.529575 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.530087 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.530872 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:08.531103 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:08.531213 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:08.533862 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:43677: BlockingRecv error: recv got EOF from 127.14.116.195:43677 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 06:49:09 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:09 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:09.165913 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:09.160473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:45279)
0619 06:49:09.160735 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:09.160749 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:09.160830 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:09.161052 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:09.161054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:09.161221 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:09.161308 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:09.161313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.162112 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.162115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:09.162656 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:09.162663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.163085 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.163087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:09.163147 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:09.163658 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:09.163669 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:09.165720 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":65,"threads_started":1}
W20260619 06:49:09.165971 24027 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:09.160554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:45279 (local address 127.14.116.254:38717)
0619 06:49:09.160830 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:09.160833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:09.160849 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:49:09.160902 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:09.160904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:09.160946 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:09.161022 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:09.161417 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.162015 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.162754 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.162993 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.163074 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:09.165849 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:45279: BlockingRecv error: recv got EOF from 127.14.116.193:45279 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":73,"threads_started":1}
Jun 19 06:49:09 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:09 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:09.337291 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:09.331047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:59589)
0619 06:49:09.331306 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:09.331319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:09.331405 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:09.331704 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:09.331706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:09.331871 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:09.331962 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:09.331967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.333217 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.333220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:09.333773 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:09.333781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.334452 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.334454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:09.334513 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:09.335113 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:09.335129 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:09.337173 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
W20260619 06:49:09.337638 24027 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:09.331120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:59589 (local address 127.14.116.254:38717)
0619 06:49:09.331302 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:09.331309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:09.331368 (+ 59us) server_negotiation.cc:407] Connection header received
0619 06:49:09.331504 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:09.331508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:09.331571 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:09.331694 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:09.332066 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.333087 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.333878 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.334335 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.334463 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:09.337379 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:59589: BlockingRecv error: recv got EOF from 127.14.116.194:59589 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 06:49:09 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:09 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:09.540608 24029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:09.534752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:60211)
0619 06:49:09.535060 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:09.535078 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:09.535211 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:09.535491 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:09.535494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:09.535720 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:09.535860 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:09.535871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.536779 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.536781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:09.537509 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:09.537518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.537944 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.537946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:09.538004 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:09.538659 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:09.538679 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:09.540497 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
W20260619 06:49:09.540779 24027 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:09.534860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:60211 (local address 127.14.116.254:38717)
0619 06:49:09.535008 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:09.535013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:09.535149 (+ 136us) server_negotiation.cc:407] Connection header received
0619 06:49:09.535322 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:09.535326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:09.535375 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:09.535468 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:09.536008 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.536683 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.537632 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:09.537852 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:09.537931 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:09.540617 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:60211: BlockingRecv error: recv got EOF from 127.14.116.195:60211 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:49:10 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:10 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:10.174322 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:10.166666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:45407)
0619 06:49:10.166918 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:10.166937 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:10.167032 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:10.167258 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:10.167260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:10.167432 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:10.167526 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:10.167531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.168249 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.168251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:10.168786 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:10.168793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.169193 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.169195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:10.169254 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:10.172386 (+ 3132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:10.172398 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:10.174194 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
W20260619 06:49:10.174530 24031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:10.166707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:45407 (local address 127.14.116.254:38717)
0619 06:49:10.166935 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:10.166938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:10.166990 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:49:10.167105 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:10.167107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:10.167149 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:10.167224 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:10.167640 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.168163 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.168881 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.169099 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.169174 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:10.174380 (+ 5206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:45407: BlockingRecv error: recv got EOF from 127.14.116.193:45407 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
Jun 19 06:49:10 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:10 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:10.343837 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:10.338027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:58453)
0619 06:49:10.338312 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:10.338330 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:10.338462 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:10.338723 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:10.338726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:10.338938 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:10.339030 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:10.339036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.339872 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.339878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:10.340443 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:10.340451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.340987 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.340991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:10.341098 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:10.341614 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:10.341626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:10.343717 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
W20260619 06:49:10.343958 24031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:10.338090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:58453 (local address 127.14.116.254:38717)
0619 06:49:10.338268 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:10.338273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:10.338409 (+ 136us) server_negotiation.cc:407] Connection header received
0619 06:49:10.338565 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:10.338568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:10.338617 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:10.338705 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:10.339151 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.339741 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.340565 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.340845 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.340947 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:10.343838 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:58453: BlockingRecv error: recv got EOF from 127.14.116.194:58453 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 06:49:10 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:10 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:10.546883 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:10.541374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:52747)
0619 06:49:10.541729 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:10.541742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:10.541877 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:10.542143 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:10.542146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:10.542355 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:10.542447 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:10.542453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.543203 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.543208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:10.543847 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:10.543856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.544341 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.544344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:10.544428 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:10.544914 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:10.544925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:10.546770 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
W20260619 06:49:10.547078 24031 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:10.541483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:52747 (local address 127.14.116.254:38717)
0619 06:49:10.541672 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:10.541676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:10.541817 (+ 141us) server_negotiation.cc:407] Connection header received
0619 06:49:10.541976 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:10.541980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:10.542034 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:10.542123 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:10.542567 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.543092 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.543997 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:10.544220 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:10.544341 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:10.546932 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:52747: BlockingRecv error: recv got EOF from 127.14.116.195:52747 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 19 06:49:11 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:11 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:11.180927 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:11.175075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:50567)
0619 06:49:11.175339 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:11.175353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:11.175435 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:11.175711 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:11.175714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:11.175903 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:11.176004 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:11.176009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.176778 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.176782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:11.177763 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:11.177774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.178286 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.178290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:11.178375 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:11.178888 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:11.178899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:11.180806 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
W20260619 06:49:11.181092 24035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:11.175143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:50567 (local address 127.14.116.254:38717)
0619 06:49:11.175375 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:11.175379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:11.175428 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:49:11.175526 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:11.175528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:11.175573 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:11.175665 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:11.176102 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.176648 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.177919 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.178158 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.178274 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:11.180950 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:50567: BlockingRecv error: recv got EOF from 127.14.116.193:50567 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":60,"threads_started":1}
Jun 19 06:49:11 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:11 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:11.350317 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:11.344563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:49819)
0619 06:49:11.344858 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:11.344876 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:11.345024 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:11.345298 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:11.345301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:11.345506 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:11.345596 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:11.345601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.346547 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.346550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:11.347386 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:11.347395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.347894 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.347897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:11.347962 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:11.348443 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:11.348455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:11.350205 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":79,"threads_started":1}
W20260619 06:49:11.350502 24035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:11.344686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:49819 (local address 127.14.116.254:38717)
0619 06:49:11.344858 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:11.344864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:11.344967 (+ 103us) server_negotiation.cc:407] Connection header received
0619 06:49:11.345125 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:11.345128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:11.345174 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:11.345279 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:11.345716 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.346395 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.347508 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.347790 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.347871 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:11.350332 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:49819: BlockingRecv error: recv got EOF from 127.14.116.194:49819 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 06:49:11 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:11 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:11.553279 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:11.547664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:44357)
0619 06:49:11.547969 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:11.547988 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:11.548117 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:11.548403 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:11.548406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:11.548612 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:11.548713 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:11.548719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.549470 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.549476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:11.550060 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:11.550068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.550533 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.550537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:11.550621 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:11.551135 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:11.551146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:11.553167 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
W20260619 06:49:11.553561 24035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:11.547757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:44357 (local address 127.14.116.254:38717)
0619 06:49:11.547926 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:11.547931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:11.548067 (+ 136us) server_negotiation.cc:407] Connection header received
0619 06:49:11.548224 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:11.548227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:11.548283 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:11.548381 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:11.548833 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.549358 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.550186 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:11.550417 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:11.550533 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:11.553302 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:44357: BlockingRecv error: recv got EOF from 127.14.116.195:44357 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 06:49:12 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:12 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:12.186846 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:12.181778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:42451)
0619 06:49:12.182037 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:12.182051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:12.182143 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:12.182364 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:12.182366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:12.182539 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:12.182629 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:12.182634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.183358 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.183361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:12.183945 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:12.183953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.184411 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.184414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:12.184483 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:12.184942 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:12.184953 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:12.186726 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
W20260619 06:49:12.186988 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:12.181812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:42451 (local address 127.14.116.254:38717)
0619 06:49:12.182049 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:12.182053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:12.182100 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:49:12.182206 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:12.182208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:12.182248 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:12.182328 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:12.182725 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.183250 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.184041 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.184267 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.184352 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:12.186857 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:42451: BlockingRecv error: recv got EOF from 127.14.116.193:42451 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
Jun 19 06:49:12 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:12 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:12.356760 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:12.351069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:52979)
0619 06:49:12.351333 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:12.351348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:12.351449 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:12.351703 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:12.351706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:12.351882 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:12.351978 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:12.351983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.352731 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.352735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:12.353398 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:12.353408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.354006 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.354012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:12.354111 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:12.354680 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:12.354691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:12.356629 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
W20260619 06:49:12.356894 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:12.351231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:52979 (local address 127.14.116.254:38717)
0619 06:49:12.351363 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:12.351366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:12.351398 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:49:12.351514 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:12.351516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:12.351556 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:12.351674 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:12.352074 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.352592 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.353557 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.353859 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.353973 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:12.356769 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:52979: BlockingRecv error: recv got EOF from 127.14.116.194:52979 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:49:12 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:12 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:12.559092 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:12.554025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:59551)
0619 06:49:12.554322 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:12.554336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:12.554434 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:12.554675 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:12.554677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:12.554855 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:12.554957 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:12.554962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.555725 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.555728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:12.556285 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:12.556292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.556708 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.556711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:12.556770 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:12.557245 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:12.557256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:12.558980 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":125,"threads_started":1}
W20260619 06:49:12.559228 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:12.554111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:59551 (local address 127.14.116.254:38717)
0619 06:49:12.554252 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:12.554257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:12.554437 (+ 180us) server_negotiation.cc:407] Connection header received
0619 06:49:12.554507 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:12.554510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:12.554561 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:12.554640 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:12.555071 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.555589 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.556392 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:12.556614 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:12.556688 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:12.559109 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:59551: BlockingRecv error: recv got EOF from 127.14.116.195:59551 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:49:13 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:13 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:13.193069 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:13.187760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:52491)
0619 06:49:13.188042 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:13.188055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:13.188139 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:13.188406 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:13.188408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:13.188584 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:13.188675 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:13.188680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.189431 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.189434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:13.189986 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:13.189994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.190414 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.190417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:13.190478 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:13.190961 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:13.190972 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:13.192960 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
W20260619 06:49:13.193310 24043 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:13.187801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:52491 (local address 127.14.116.254:38717)
0619 06:49:13.188102 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:13.188105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:13.188118 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:49:13.188229 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:13.188231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:13.188277 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:13.188364 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:13.188792 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.189327 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.190086 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.190313 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.190387 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:13.193092 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:52491: BlockingRecv error: recv got EOF from 127.14.116.193:52491 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":86,"threads_started":1}
Jun 19 06:49:13 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:13 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:13.362622 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:13.357503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:55905)
0619 06:49:13.357771 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:13.357795 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:13.357889 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:13.358119 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:13.358121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:13.358300 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:13.358390 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:13.358394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.359130 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.359133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:13.359735 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:13.359745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.360165 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.360168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:13.360227 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:13.360757 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:13.360769 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:13.362511 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":107,"threads_started":1}
W20260619 06:49:13.362819 24043 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:13.357609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:55905 (local address 127.14.116.254:38717)
0619 06:49:13.357754 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:13.357758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:13.357861 (+ 103us) server_negotiation.cc:407] Connection header received
0619 06:49:13.357970 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:13.357972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:13.358022 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:13.358084 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:13.358499 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.359022 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.359849 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.360069 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.360148 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:13.362652 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:55905: BlockingRecv error: recv got EOF from 127.14.116.194:55905 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:49:13 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:13 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.195@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:13.565467 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:13.559961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.195:56803)
0619 06:49:13.560228 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:13.560241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:13.560341 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:13.560605 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:13.560607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:13.560794 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:13.560906 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:13.560912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.561687 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.561693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:13.562297 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:13.562306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.562767 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.562770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:13.562862 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:13.563412 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:13.563424 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:13.565267 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":110,"threads_started":1}
W20260619 06:49:13.565503 24043 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:13.560047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:56803 (local address 127.14.116.254:38717)
0619 06:49:13.560175 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:13.560179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:13.560346 (+ 167us) server_negotiation.cc:407] Connection header received
0619 06:49:13.560420 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:13.560422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:13.560495 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:13.560574 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:13.561038 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.561570 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.562425 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:13.562650 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:13.562766 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:13.565397 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.195:56803: BlockingRecv error: recv got EOF from 127.14.116.195:56803 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 06:49:14 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:14 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.193@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:14.198954 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:14.193922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.193:42409)
0619 06:49:14.194206 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:14.194220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:14.194327 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:14.194562 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:14.194564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:14.194739 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:14.194831 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:14.194836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.195549 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.195551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:14.196191 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:14.196200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.196622 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.196624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:14.196683 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:14.197144 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:14.197154 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:14.198833 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":105,"threads_started":1}
W20260619 06:49:14.199079 24047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:14.193995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:42409 (local address 127.14.116.254:38717)
0619 06:49:14.194236 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:14.194239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:14.194304 (+ 65us) server_negotiation.cc:407] Connection header received
0619 06:49:14.194404 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:14.194407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:14.194454 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:14.194529 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:14.194927 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.195453 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.196300 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.196530 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.196605 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:14.198960 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.193:42409: BlockingRecv error: recv got EOF from 127.14.116.193:42409 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
Jun 19 06:49:14 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:49:14 dist-test-slave-9x3b krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.116.194@KRBTEST.COM for oryx/127.14.116.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:49:14.368634 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:14.363522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38717 (local address 127.14.116.194:41221)
0619 06:49:14.363768 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:14.363797 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:14.363874 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:14.364077 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:14.364079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:14.364253 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:14.364343 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:14.364348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.365156 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.365159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:14.365763 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:14.365772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.366171 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.366174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:14.366230 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:14.366765 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:14.366776 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:14.368510 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.116.254:38717: Server oryx/127.14.116.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":117,"threads_started":1}
W20260619 06:49:14.368810 24047 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:14.363718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:41221 (local address 127.14.116.254:38717)
0619 06:49:14.363859 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:14.363863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:14.363875 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:49:14.363940 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:14.363942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:14.363982 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:14.364043 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:14.364438 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.365061 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.365861 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.366081 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.366158 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:14.368674 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.116.194:41221: BlockingRecv error: recv got EOF from 127.14.116.194:41221 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:49:14.463701 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 23571
I20260619 06:49:14.469549 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 23722
I20260619 06:49:14.475924 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 23856
I20260619 06:49:14.480819 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 23501
2026-06-19T06:49:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17828 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T06:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:49:14Z Disabled control of system clock
I20260619 06:49:14.510285 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:35815
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:49:14.616426 24052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:14.616670 24052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:14.616704 24052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:14.620604 24052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:49:14.620659 24052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:14.620675 24052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:14.620692 24052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:49:14.620707 24052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:49:14.625314 24052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:35815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:14.626209 24052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:14.627487 24052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:14.635720 24057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:14.635797 24052 server_base.cc:1034] running on GCE node
W20260619 06:49:14.635768 24058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:14.635738 24060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:14.636547 24052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:14.637558 24052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:14.638749 24052 hybrid_clock.cc:648] HybridClock initialized: now 1781851754638718 us; error 44 us; skew 500 ppm
I20260619 06:49:14.640552 24052 webserver.cc:460] Webserver started at http://127.14.116.254:45249/ using document root <none> and password file <none>
I20260619 06:49:14.641112 24052 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:14.641204 24052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:14.641431 24052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:14.643394 24052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "2a713adf89654f70adfef37580718258"
format_stamp: "Formatted at 2026-06-19 06:49:14 on dist-test-slave-9x3b"
server_key: "9a1c4057edd06b396b05cfe67d511298"
server_key_iv: "7c73eb3807de517810dceb5e808eeb12"
server_key_version: "encryptionkey@0"
I20260619 06:49:14.644006 24052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "2a713adf89654f70adfef37580718258"
format_stamp: "Formatted at 2026-06-19 06:49:14 on dist-test-slave-9x3b"
server_key: "9a1c4057edd06b396b05cfe67d511298"
server_key_iv: "7c73eb3807de517810dceb5e808eeb12"
server_key_version: "encryptionkey@0"
I20260619 06:49:14.647478 24052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:49:14.649781 24066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:14.650957 24052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:49:14.651094 24052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "2a713adf89654f70adfef37580718258"
format_stamp: "Formatted at 2026-06-19 06:49:14 on dist-test-slave-9x3b"
server_key: "9a1c4057edd06b396b05cfe67d511298"
server_key_iv: "7c73eb3807de517810dceb5e808eeb12"
server_key_version: "encryptionkey@0"
I20260619 06:49:14.651206 24052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:14.669385 24052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:14.670075 24052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:14.670274 24052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:14.676884 24052 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:35815
I20260619 06:49:14.676892 24118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:35815 every 8 connection(s)
I20260619 06:49:14.677826 24052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:49:14.680831 24119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:49:14.685555 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 24052
I20260619 06:49:14.685693 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:49:14.685972 14803 external_mini_cluster.cc:1428] Setting key b0366a7dc7fa4113412fe5cc577b38b2
I20260619 06:49:14.686359 24119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258: Bootstrap starting.
I20260619 06:49:14.688792 24119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258: Neither blocks nor log segments found. Creating new log.
I20260619 06:49:14.689493 24119 log.cc:824] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258: Log is configured to *not* fsync() on all Append() calls
I20260619 06:49:14.691274 24119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258: No bootstrap required, opened a new log
I20260619 06:49:14.694680 24119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a713adf89654f70adfef37580718258" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35815 } }
I20260619 06:49:14.694912 24119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:49:14.694994 24119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a713adf89654f70adfef37580718258, State: Initialized, Role: FOLLOWER
I20260619 06:49:14.695024 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:14.687339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.14.116.254:35815)
0619 06:49:14.688029 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:14.688043 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:14.688077 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:49:14.688657 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:14.688673 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:14.689115 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:14.689447 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:14.689856 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.690770 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.691816 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.692280 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.692463 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:14.693320 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:14.693324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:14.693336 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:49:14.693699 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:14.693760 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:49:14.693786 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:14.693897 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:14.694011 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 06:49:14.694279 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":126,"threads_started":1}
I20260619 06:49:14.695493 24119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a713adf89654f70adfef37580718258" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35815 } }
I20260619 06:49:14.695662 24119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:49:14.695726 24119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:49:14.695814 24119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:49:14.696668 24119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a713adf89654f70adfef37580718258" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35815 } }
I20260619 06:49:14.697031 24119 leader_election.cc:302] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a713adf89654f70adfef37580718258; no voters:
I20260619 06:49:14.697355 24119 leader_election.cc:288] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:49:14.697470 24124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:49:14.697717 24124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [term 1 LEADER]: Becoming Leader. State: Replica: 2a713adf89654f70adfef37580718258, State: Running, Role: LEADER
I20260619 06:49:14.698067 24124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a713adf89654f70adfef37580718258" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35815 } }
I20260619 06:49:14.698524 24119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:49:14.699637 24125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a713adf89654f70adfef37580718258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a713adf89654f70adfef37580718258" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35815 } } }
I20260619 06:49:14.699764 24125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [sys.catalog]: This master's current role is: LEADER
I20260619 06:49:14.700052 24126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a713adf89654f70adfef37580718258. Latest consensus state: current_term: 1 leader_uuid: "2a713adf89654f70adfef37580718258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a713adf89654f70adfef37580718258" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35815 } } }
I20260619 06:49:14.700202 24126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258 [sys.catalog]: This master's current role is: LEADER
I20260619 06:49:14.700106 24133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:49:14.703233 24133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:49:14.708273 24133 catalog_manager.cc:1347] Generated new cluster ID: 19cc9125d7c44bc0b4e23d71aac7fdca
I20260619 06:49:14.708333 24133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:49:14.730832 24133 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:49:14.732016 24133 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:49:14.736901 24133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a713adf89654f70adfef37580718258: Generated new TSK 0
I20260619 06:49:14.737550 24133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:49:14.740844 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35815
--builtin_ntp_servers=127.14.116.212:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 06:49:14.849400 24143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:14.849625 24143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:14.849686 24143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:14.853655 24143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:14.853703 24143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:14.853811 24143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:49:14.858606 24143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:14.859627 24143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:14.860963 24143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:14.870168 24149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:14.870252 24151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:14.870193 24148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:14.870385 24143 server_base.cc:1034] running on GCE node
I20260619 06:49:14.870956 24143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:14.871548 24143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:14.872704 24143 hybrid_clock.cc:648] HybridClock initialized: now 1781851754872692 us; error 23 us; skew 500 ppm
I20260619 06:49:14.874482 24143 webserver.cc:460] Webserver started at http://127.14.116.193:34829/ using document root <none> and password file <none>
I20260619 06:49:14.875025 24143 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:14.875085 24143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:14.875336 24143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:14.877352 24143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "add55d85efb441f0941acb3dc59106aa"
format_stamp: "Formatted at 2026-06-19 06:49:14 on dist-test-slave-9x3b"
server_key: "3fc150cc18cd4ed1d2125ddf7f5da2b8"
server_key_iv: "cf0702d1c4ba1fb3044adbfa509f71c1"
server_key_version: "encryptionkey@0"
I20260619 06:49:14.877900 24143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "add55d85efb441f0941acb3dc59106aa"
format_stamp: "Formatted at 2026-06-19 06:49:14 on dist-test-slave-9x3b"
server_key: "3fc150cc18cd4ed1d2125ddf7f5da2b8"
server_key_iv: "cf0702d1c4ba1fb3044adbfa509f71c1"
server_key_version: "encryptionkey@0"
I20260619 06:49:14.881438 24143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:49:14.883884 24157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:14.885056 24143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:49:14.885174 24143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "add55d85efb441f0941acb3dc59106aa"
format_stamp: "Formatted at 2026-06-19 06:49:14 on dist-test-slave-9x3b"
server_key: "3fc150cc18cd4ed1d2125ddf7f5da2b8"
server_key_iv: "cf0702d1c4ba1fb3044adbfa509f71c1"
server_key_version: "encryptionkey@0"
I20260619 06:49:14.885280 24143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:14.895272 24143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:14.895891 24143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:14.896077 24143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:14.896612 24143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:49:14.897470 24143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:49:14.897514 24143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:14.897584 24143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:49:14.897615 24143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:14.906081 24143 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:41011
I20260619 06:49:14.906101 24270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:41011 every 8 connection(s)
I20260619 06:49:14.906924 24143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:49:14.915354 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:14.908633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:54095 (local address 127.14.116.254:35815)
0619 06:49:14.908799 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:14.908805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:14.909788 (+ 983us) server_negotiation.cc:407] Connection header received
0619 06:49:14.911080 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:14.911082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:14.911127 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:14.911199 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:14.912116 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.912642 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.913693 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.913917 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.913993 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:14.914968 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:14.914971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:14.914973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:49:14.914986 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:14.915004 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:49:14.915011 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:14.915068 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:14.915196 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 06:49:14.915242 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:49:14.916201 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:14.909018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35815 (local address 127.14.116.193:54095)
0619 06:49:14.909635 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:14.909674 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:14.910876 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:14.911361 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:14.911367 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:14.911448 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:14.911988 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:14.912001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.912744 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.912746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:14.913592 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:14.913600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:14.914049 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:14.914052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:14.914180 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:14.914735 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:49:14.914736 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:14.914773 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:49:14.914830 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:49:14.915081 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:49:14.915087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:49:14.915091 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:49:14.915179 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:49:14.915378 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260619 06:49:14.916857 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 24143
I20260619 06:49:14.916989 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:49:14.917317 14803 external_mini_cluster.cc:1428] Setting key 15eb7ae632e764fbf83877f555778892
I20260619 06:49:14.917543 24271 heartbeater.cc:347] Connected to a master server at 127.14.116.254:35815
I20260619 06:49:14.917804 24271 heartbeater.cc:464] Registering TS with master...
I20260619 06:49:14.918370 24271 heartbeater.cc:511] Master 127.14.116.254:35815 requested a full tablet report, sending...
I20260619 06:49:14.919342 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35815
--builtin_ntp_servers=127.14.116.212:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:49:14.919953 24083 ts_manager.cc:194] Registered new tserver with Master: add55d85efb441f0941acb3dc59106aa (127.14.116.193:41011)
I20260619 06:49:14.921268 24083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.193:54095
W20260619 06:49:15.026064 24274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:15.026365 24274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:15.026444 24274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:15.030402 24274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:15.030500 24274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:15.030588 24274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:49:15.035171 24274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:15.036199 24274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:15.037521 24274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:15.046556 24282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:15.046633 24279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:15.046635 24280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:15.046933 24274 server_base.cc:1034] running on GCE node
I20260619 06:49:15.047327 24274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:15.047982 24274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:15.049178 24274 hybrid_clock.cc:648] HybridClock initialized: now 1781851755049166 us; error 32 us; skew 500 ppm
I20260619 06:49:15.050855 24274 webserver.cc:460] Webserver started at http://127.14.116.194:43981/ using document root <none> and password file <none>
I20260619 06:49:15.051419 24274 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:15.051517 24274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:15.051786 24274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:15.053721 24274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "961da931b09e4735bc208e0dc3ddd175"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "27e8a6026ef1a9c25b12c4a32b0b6304"
server_key_iv: "2d946caeda69338fda540dcb39c6201d"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.054370 24274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "961da931b09e4735bc208e0dc3ddd175"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "27e8a6026ef1a9c25b12c4a32b0b6304"
server_key_iv: "2d946caeda69338fda540dcb39c6201d"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.057960 24274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:49:15.060307 24288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.061555 24274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:49:15.061709 24274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "961da931b09e4735bc208e0dc3ddd175"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "27e8a6026ef1a9c25b12c4a32b0b6304"
server_key_iv: "2d946caeda69338fda540dcb39c6201d"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.061833 24274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:15.090734 24274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:15.091419 24274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:15.091668 24274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:15.092255 24274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:49:15.093174 24274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:49:15.093250 24274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.093317 24274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:49:15.093351 24274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.101986 24274 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:45957
I20260619 06:49:15.102005 24401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:45957 every 8 connection(s)
I20260619 06:49:15.102864 24274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:49:15.105898 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 24274
I20260619 06:49:15.106012 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:49:15.106321 14803 external_mini_cluster.cc:1428] Setting key 0dc28c2844db83e87138ee890121492e
I20260619 06:49:15.108402 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35815
--builtin_ntp_servers=127.14.116.212:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:49:15.113189 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:15.104693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:52129 (local address 127.14.116.254:35815)
0619 06:49:15.104852 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:15.104857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:15.105972 (+ 1115us) server_negotiation.cc:407] Connection header received
0619 06:49:15.107681 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:15.107685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:15.107738 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:15.107812 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:15.108741 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:15.109582 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:15.111255 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:15.111468 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:15.111584 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:15.112694 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:15.112698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:15.112700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:49:15.112719 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:15.112737 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:49:15.112745 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:15.112823 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:15.112997 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 06:49:15.113057 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:49:15.113993 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:15.105094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35815 (local address 127.14.116.194:52129)
0619 06:49:15.105765 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:15.105825 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:15.107427 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:15.107987 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:15.107994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:15.108078 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:15.108580 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:15.108599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:15.109730 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:15.109736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:15.111138 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:15.111147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:15.111654 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:15.111659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:15.111817 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:15.112436 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:49:15.112438 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:15.112477 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:49:15.112535 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:49:15.112837 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:49:15.112848 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:49:15.112855 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:49:15.112993 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 06:49:15.113259 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":98,"threads_started":1}
I20260619 06:49:15.115253 24402 heartbeater.cc:347] Connected to a master server at 127.14.116.254:35815
I20260619 06:49:15.115478 24402 heartbeater.cc:464] Registering TS with master...
I20260619 06:49:15.115999 24402 heartbeater.cc:511] Master 127.14.116.254:35815 requested a full tablet report, sending...
I20260619 06:49:15.117089 24083 ts_manager.cc:194] Registered new tserver with Master: 961da931b09e4735bc208e0dc3ddd175 (127.14.116.194:45957)
I20260619 06:49:15.117934 24083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.194:52129
W20260619 06:49:15.218710 24405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:15.218956 24405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:15.218992 24405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:15.222854 24405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:15.222903 24405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:15.222954 24405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:49:15.227551 24405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:15.228488 24405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:15.229723 24405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:15.238608 24413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:15.238598 24410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:15.238600 24411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:15.238929 24405 server_base.cc:1034] running on GCE node
I20260619 06:49:15.239343 24405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:15.239930 24405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:15.241110 24405 hybrid_clock.cc:648] HybridClock initialized: now 1781851755241089 us; error 46 us; skew 500 ppm
I20260619 06:49:15.246165 24405 webserver.cc:460] Webserver started at http://127.14.116.195:36009/ using document root <none> and password file <none>
I20260619 06:49:15.246698 24405 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:15.246788 24405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:15.246991 24405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:15.248909 24405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "c5b37b815f0344179e9b10016d71841a"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "68e184d702f53a6d158c0514646206e8"
server_key_iv: "e59c291f089cc91d801cceb28bd9dbe0"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.249456 24405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "c5b37b815f0344179e9b10016d71841a"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "68e184d702f53a6d158c0514646206e8"
server_key_iv: "e59c291f089cc91d801cceb28bd9dbe0"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.252854 24405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:49:15.255259 24419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.256668 24405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:49:15.256811 24405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "c5b37b815f0344179e9b10016d71841a"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "68e184d702f53a6d158c0514646206e8"
server_key_iv: "e59c291f089cc91d801cceb28bd9dbe0"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.256925 24405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:15.266952 24405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:15.267539 24405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:15.267748 24405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:15.268296 24405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:49:15.269160 24405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:49:15.269236 24405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.269313 24405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:49:15.269371 24405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.277544 24405 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:43183
I20260619 06:49:15.277562 24532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:43183 every 8 connection(s)
I20260619 06:49:15.278368 24405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:49:15.285225 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 24405
I20260619 06:49:15.285356 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:49:15.285630 14803 external_mini_cluster.cc:1428] Setting key 42cbaefd28df10473fa62f3e4e482cc2
I20260619 06:49:15.287523 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:15.280082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:60279 (local address 127.14.116.254:35815)
0619 06:49:15.280214 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:15.280218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:15.281191 (+ 973us) server_negotiation.cc:407] Connection header received
0619 06:49:15.282667 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:15.282671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:15.282729 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:15.282849 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:15.283723 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:15.284250 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:15.285482 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:15.285729 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:15.285857 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:15.287053 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:15.287062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:15.287066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:49:15.287079 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:15.287102 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:49:15.287112 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:15.287184 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:15.287329 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 06:49:15.287383 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:49:15.288349 24535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:15.280455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35815 (local address 127.14.116.195:60279)
0619 06:49:15.281048 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:15.281086 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:15.282329 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:15.283017 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:15.283023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:15.283102 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:15.283543 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:15.283554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:15.284407 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:15.284410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:15.285271 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:15.285281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:15.285887 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:15.285891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:15.286094 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:15.286724 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:49:15.286726 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:15.286777 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:49:15.286849 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:49:15.287195 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:49:15.287202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:49:15.287206 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:49:15.287325 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 06:49:15.287525 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
I20260619 06:49:15.289395 24533 heartbeater.cc:347] Connected to a master server at 127.14.116.254:35815
I20260619 06:49:15.289674 24533 heartbeater.cc:464] Registering TS with master...
I20260619 06:49:15.290184 24533 heartbeater.cc:511] Master 127.14.116.254:35815 requested a full tablet report, sending...
I20260619 06:49:15.291087 24083 ts_manager.cc:194] Registered new tserver with Master: c5b37b815f0344179e9b10016d71841a (127.14.116.195:43183)
I20260619 06:49:15.291710 24083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.195:60279
I20260619 06:49:15.299477 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:49:15.302330 24122 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:15.301381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.14.116.254:35815)
0619 06:49:15.301552 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:15.301557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:15.301695 (+ 138us) server_negotiation.cc:407] Connection header received
0619 06:49:15.301836 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:15.301839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:15.301879 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:15.301972 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:15.302206 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56458: BlockingRecv error: recv got EOF from 127.0.0.1:56458 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:49:15.302462 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 24143
I20260619 06:49:15.307915 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 24274
I20260619 06:49:15.312829 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 24405
I20260619 06:49:15.317767 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 24052
2026-06-19T06:49:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (835 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T06:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:49:15Z Disabled control of system clock
I20260619 06:49:15.345358 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46465
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:46465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:49:15.448855 24544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:15.449100 24544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:15.449133 24544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:15.453073 24544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:49:15.453122 24544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:15.453138 24544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:15.453154 24544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:49:15.453169 24544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:49:15.457681 24544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:46465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:15.458572 24544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:15.459896 24544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:15.467941 24550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:15.467937 24552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:15.467937 24549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:15.468189 24544 server_base.cc:1034] running on GCE node
I20260619 06:49:15.468747 24544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:15.469813 24544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:15.471030 24544 hybrid_clock.cc:648] HybridClock initialized: now 1781851755471009 us; error 41 us; skew 500 ppm
I20260619 06:49:15.472858 24544 webserver.cc:460] Webserver started at http://127.14.116.254:38173/ using document root <none> and password file <none>
I20260619 06:49:15.473412 24544 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:15.473506 24544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:15.473740 24544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:15.475777 24544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "500a464172b0485392d85d11337096b7"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "568d1042fcb526fb7e8b80aedc1cec6c"
server_key_iv: "fa5280134663ef389951ff48a9a1772f"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.476326 24544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "500a464172b0485392d85d11337096b7"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "568d1042fcb526fb7e8b80aedc1cec6c"
server_key_iv: "fa5280134663ef389951ff48a9a1772f"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.480057 24544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:49:15.482318 24558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.483706 24544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:49:15.483834 24544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "500a464172b0485392d85d11337096b7"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "568d1042fcb526fb7e8b80aedc1cec6c"
server_key_iv: "fa5280134663ef389951ff48a9a1772f"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.483953 24544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:15.488309 24544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:15.488888 24544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:15.489069 24544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:15.495455 24544 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:46465
I20260619 06:49:15.495481 24610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:46465 every 8 connection(s)
I20260619 06:49:15.496383 24544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:49:15.499287 24611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:49:15.500165 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 24544
I20260619 06:49:15.500283 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:49:15.500584 14803 external_mini_cluster.cc:1428] Setting key 7ca73a68d69f0cd154a1aa84f636c646
I20260619 06:49:15.505359 24611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7: Bootstrap starting.
I20260619 06:49:15.505939 24614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:15.501895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.14.116.254:46465)
0619 06:49:15.502502 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:15.502517 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:15.502552 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:49:15.503177 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:15.503192 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:15.503521 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:15.503920 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:15.504674 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:15.504679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:15.504690 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:49:15.504700 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:15.504752 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:49:15.504767 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:15.504919 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:15.504988 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 06:49:15.505218 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260619 06:49:15.507592 24611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7: Neither blocks nor log segments found. Creating new log.
I20260619 06:49:15.508419 24611 log.cc:824] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:49:15.510092 24611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7: No bootstrap required, opened a new log
I20260619 06:49:15.513437 24611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500a464172b0485392d85d11337096b7" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46465 } }
I20260619 06:49:15.513715 24611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:49:15.513775 24611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500a464172b0485392d85d11337096b7, State: Initialized, Role: FOLLOWER
I20260619 06:49:15.514207 24611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500a464172b0485392d85d11337096b7" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46465 } }
I20260619 06:49:15.514349 24611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:49:15.514441 24611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:49:15.514540 24611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:49:15.515359 24611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500a464172b0485392d85d11337096b7" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46465 } }
I20260619 06:49:15.515751 24611 leader_election.cc:302] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500a464172b0485392d85d11337096b7; no voters:
I20260619 06:49:15.516032 24611 leader_election.cc:288] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:49:15.516151 24616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:49:15.516376 24616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [term 1 LEADER]: Becoming Leader. State: Replica: 500a464172b0485392d85d11337096b7, State: Running, Role: LEADER
I20260619 06:49:15.516657 24616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500a464172b0485392d85d11337096b7" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46465 } }
I20260619 06:49:15.517189 24611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:49:15.518162 24617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500a464172b0485392d85d11337096b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500a464172b0485392d85d11337096b7" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46465 } } }
I20260619 06:49:15.518293 24617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:49:15.518538 24618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500a464172b0485392d85d11337096b7. Latest consensus state: current_term: 1 leader_uuid: "500a464172b0485392d85d11337096b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500a464172b0485392d85d11337096b7" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46465 } } }
I20260619 06:49:15.518623 24618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:49:15.518748 24625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:49:15.522059 24625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:49:15.527751 24625 catalog_manager.cc:1347] Generated new cluster ID: 3ef0bc5e04be43fbadd81019edbe65da
I20260619 06:49:15.527815 24625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:49:15.533716 24625 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:49:15.534013 24625 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:49:15.538203 24625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500a464172b0485392d85d11337096b7: Generated new TSK 0
I20260619 06:49:15.538846 24625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:49:15.545889 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46465
--builtin_ntp_servers=127.14.116.212:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:49:15.654755 24635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:15.655045 24635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:15.655119 24635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:15.659179 24635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:15.659263 24635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:15.659363 24635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:49:15.664189 24635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:15.665211 24635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:15.666543 24635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:15.675714 24641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:15.675757 24643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:15.675714 24640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:15.675932 24635 server_base.cc:1034] running on GCE node
I20260619 06:49:15.676357 24635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:15.676976 24635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:15.678160 24635 hybrid_clock.cc:648] HybridClock initialized: now 1781851755678148 us; error 23 us; skew 500 ppm
I20260619 06:49:15.679937 24635 webserver.cc:460] Webserver started at http://127.14.116.193:37703/ using document root <none> and password file <none>
I20260619 06:49:15.680506 24635 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:15.680600 24635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:15.680828 24635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:15.682866 24635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "390ae2145e2f4efda554e77714834751"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "f6ddeef385cd10791a8faf33387fbe81"
server_key_iv: "ca56c6976d0695274fe0026446374d32"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.683439 24635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "390ae2145e2f4efda554e77714834751"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "f6ddeef385cd10791a8faf33387fbe81"
server_key_iv: "ca56c6976d0695274fe0026446374d32"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.687253 24635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:49:15.689822 24649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.691285 24635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:49:15.691421 24635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "390ae2145e2f4efda554e77714834751"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "f6ddeef385cd10791a8faf33387fbe81"
server_key_iv: "ca56c6976d0695274fe0026446374d32"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.691577 24635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:15.696070 24635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:15.696650 24635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:15.696847 24635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:15.697365 24635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:49:15.698223 24635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:49:15.698303 24635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.698369 24635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:49:15.698398 24635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.706772 24762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:45857 every 8 connection(s)
I20260619 06:49:15.706821 24635 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:45857
I20260619 06:49:15.707758 24635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:49:15.712182 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 24635
I20260619 06:49:15.712371 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:49:15.712782 14803 external_mini_cluster.cc:1428] Setting key dcf7c4d9afe73a5330a58519125594ab
I20260619 06:49:15.714162 24614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:15.709512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:36603 (local address 127.14.116.254:46465)
0619 06:49:15.709679 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:15.709684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:15.710665 (+ 981us) server_negotiation.cc:407] Connection header received
0619 06:49:15.711950 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:15.711954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:15.711999 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:15.712106 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:15.713268 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:15.713272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:15.713274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:49:15.713286 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:15.713308 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:49:15.713316 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:15.713404 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:15.713950 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 06:49:15.714021 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:49:15.714723 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:15.709915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46465 (local address 127.14.116.193:36603)
0619 06:49:15.710528 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:15.710565 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:15.711708 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:15.712241 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:15.712248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:15.712368 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:15.712996 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:49:15.712998 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:15.713035 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:49:15.713097 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:49:15.713418 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:49:15.713427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:49:15.713433 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:49:15.713595 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:49:15.713786 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260619 06:49:15.714829 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46465
--builtin_ntp_servers=127.14.116.212:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 06:49:15.716009 24763 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46465
I20260619 06:49:15.716236 24763 heartbeater.cc:464] Registering TS with master...
I20260619 06:49:15.716661 24763 heartbeater.cc:511] Master 127.14.116.254:46465 requested a full tablet report, sending...
I20260619 06:49:15.718130 24575 ts_manager.cc:194] Registered new tserver with Master: 390ae2145e2f4efda554e77714834751 (127.14.116.193:45857)
W20260619 06:49:15.822784 24766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:15.823010 24766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:15.823046 24766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:15.826973 24766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:15.827021 24766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:15.827101 24766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:49:15.831730 24766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:15.832655 24766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:15.833932 24766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:15.843077 24774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:15.843042 24771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:15.843245 24766 server_base.cc:1034] running on GCE node
W20260619 06:49:15.843247 24772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:15.843775 24766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:15.844398 24766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:15.845592 24766 hybrid_clock.cc:648] HybridClock initialized: now 1781851755845580 us; error 22 us; skew 500 ppm
I20260619 06:49:15.847393 24766 webserver.cc:460] Webserver started at http://127.14.116.194:45017/ using document root <none> and password file <none>
I20260619 06:49:15.848016 24766 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:15.848109 24766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:15.848325 24766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:15.850282 24766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "2d08217067f144319e5699912ba30fcc"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "cffd4f9f386d0d7bbe0062d074e912bb"
server_key_iv: "c63aff4d20b8da6259486ca27fa707df"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.850831 24766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "2d08217067f144319e5699912ba30fcc"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "cffd4f9f386d0d7bbe0062d074e912bb"
server_key_iv: "c63aff4d20b8da6259486ca27fa707df"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.854300 24766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:49:15.856758 24780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.857893 24766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:49:15.858026 24766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "2d08217067f144319e5699912ba30fcc"
format_stamp: "Formatted at 2026-06-19 06:49:15 on dist-test-slave-9x3b"
server_key: "cffd4f9f386d0d7bbe0062d074e912bb"
server_key_iv: "c63aff4d20b8da6259486ca27fa707df"
server_key_version: "encryptionkey@0"
I20260619 06:49:15.858136 24766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:15.862536 24766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:15.863099 24766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:15.863296 24766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:15.863812 24766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:49:15.864652 24766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:49:15.864725 24766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.864789 24766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:49:15.864845 24766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:15.873337 24893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:39601 every 8 connection(s)
I20260619 06:49:15.873368 24766 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:39601
I20260619 06:49:15.874161 24766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:49:15.879837 24614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:15.875915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:54185 (local address 127.14.116.254:46465)
0619 06:49:15.876073 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:15.876078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:15.877089 (+ 1011us) server_negotiation.cc:407] Connection header received
0619 06:49:15.878352 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:15.878354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:15.878391 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:15.878467 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:15.879434 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:15.879437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:15.879439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:49:15.879450 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:15.879467 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:49:15.879474 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:15.879546 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:15.879695 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 06:49:15.879730 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:49:15.880656 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:15.876329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46465 (local address 127.14.116.194:54185)
0619 06:49:15.876949 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:15.876985 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:15.878147 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:15.878614 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:15.878620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:15.878701 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:15.879204 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:49:15.879205 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:15.879242 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:49:15.879303 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:49:15.879529 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:49:15.879533 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:49:15.879536 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:49:15.879705 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 06:49:15.879887 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260619 06:49:15.881286 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 24766
I20260619 06:49:15.881424 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:49:15.881747 14803 external_mini_cluster.cc:1428] Setting key e5d765b512472751942a48fa5ec33891
I20260619 06:49:15.881744 24894 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46465
I20260619 06:49:15.881960 24894 heartbeater.cc:464] Registering TS with master...
I20260619 06:49:15.882390 24894 heartbeater.cc:511] Master 127.14.116.254:46465 requested a full tablet report, sending...
I20260619 06:49:15.883576 24575 ts_manager.cc:194] Registered new tserver with Master: 2d08217067f144319e5699912ba30fcc (127.14.116.194:39601)
I20260619 06:49:15.883980 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46465
--builtin_ntp_servers=127.14.116.212:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:49:15.991091 24897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:15.991348 24897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:15.991387 24897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:15.995345 24897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:15.995394 24897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:15.995451 24897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:49:16.000155 24897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:16.001183 24897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:16.002517 24897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:49:16.011926 24897 server_base.cc:1034] running on GCE node
W20260619 06:49:16.011888 24902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:16.011910 24903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:16.011888 24905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:16.012519 24897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:16.013073 24897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:16.014242 24897 hybrid_clock.cc:648] HybridClock initialized: now 1781851756014219 us; error 28 us; skew 500 ppm
I20260619 06:49:16.016052 24897 webserver.cc:460] Webserver started at http://127.14.116.195:35267/ using document root <none> and password file <none>
I20260619 06:49:16.016597 24897 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:16.016657 24897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:16.016887 24897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:16.019029 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "b2de84b223464fdb967423f40a7cd130"
format_stamp: "Formatted at 2026-06-19 06:49:16 on dist-test-slave-9x3b"
server_key: "4b107dab640f1784eb3d3c90a46a8c9a"
server_key_iv: "7f2b309ceb8988e065179872e839b38e"
server_key_version: "encryptionkey@0"
I20260619 06:49:16.019572 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "b2de84b223464fdb967423f40a7cd130"
format_stamp: "Formatted at 2026-06-19 06:49:16 on dist-test-slave-9x3b"
server_key: "4b107dab640f1784eb3d3c90a46a8c9a"
server_key_iv: "7f2b309ceb8988e065179872e839b38e"
server_key_version: "encryptionkey@0"
I20260619 06:49:16.023109 24897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:49:16.025539 24911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:16.026690 24897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:49:16.026809 24897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "b2de84b223464fdb967423f40a7cd130"
format_stamp: "Formatted at 2026-06-19 06:49:16 on dist-test-slave-9x3b"
server_key: "4b107dab640f1784eb3d3c90a46a8c9a"
server_key_iv: "7f2b309ceb8988e065179872e839b38e"
server_key_version: "encryptionkey@0"
I20260619 06:49:16.026909 24897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:16.031414 24897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:16.032068 24897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:16.032243 24897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:16.032752 24897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:49:16.033610 24897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:49:16.033656 24897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:16.033722 24897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:49:16.033759 24897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:16.042624 24897 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:41825
I20260619 06:49:16.042642 25024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:41825 every 8 connection(s)
I20260619 06:49:16.043488 24897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:49:16.049394 24614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:16.045254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:54385 (local address 127.14.116.254:46465)
0619 06:49:16.045409 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:16.045413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:16.046430 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 06:49:16.047719 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:16.047723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:16.047764 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:16.047843 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:16.048917 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:16.048920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:16.048923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:49:16.048943 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:16.048963 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:49:16.048974 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:16.049076 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:16.049221 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 06:49:16.049263 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:49:16.050100 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 24897
I20260619 06:49:16.050207 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:49:16.050388 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:16.045688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46465 (local address 127.14.116.195:54385)
0619 06:49:16.046296 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:16.046333 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:16.047470 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:16.047995 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:16.048001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:16.048084 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:16.048668 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:49:16.048670 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:16.048706 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:49:16.048769 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:49:16.049072 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:49:16.049080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:49:16.049087 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:49:16.049262 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 06:49:16.049498 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260619 06:49:16.050530 14803 external_mini_cluster.cc:1428] Setting key 613a57814e253daec11716ba8e40a6b0
I20260619 06:49:16.051509 25025 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46465
I20260619 06:49:16.051743 25025 heartbeater.cc:464] Registering TS with master...
I20260619 06:49:16.052124 25025 heartbeater.cc:511] Master 127.14.116.254:46465 requested a full tablet report, sending...
I20260619 06:49:16.053063 24574 ts_manager.cc:194] Registered new tserver with Master: b2de84b223464fdb967423f40a7cd130 (127.14.116.195:41825)
I20260619 06:49:16.064096 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:49:16.066658 24614 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:49:16.065952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.14.116.254:46465)
0619 06:49:16.066069 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:16.066073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:16.066214 (+ 141us) server_negotiation.cc:407] Connection header received
0619 06:49:16.066309 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:16.066311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:16.066340 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:16.066407 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:16.066564 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50438: BlockingRecv error: recv got EOF from 127.0.0.1:50438 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:49:16.066947 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 24635
I20260619 06:49:16.072520 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 24766
I20260619 06:49:16.077720 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 24897
I20260619 06:49:16.082590 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 24544
2026-06-19T06:49:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (765 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:49:16 dist-test-slave-9x3b krb5kdc[25035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:49:16 dist-test-slave-9x3b krb5kdc[25035](info): set up 2 sockets
Jun 19 06:49:16 dist-test-slave-9x3b krb5kdc[25035](info): commencing operation
krb5kdc: starting...
W20260619 06:49:17.124621 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:49:17 dist-test-slave-9x3b krb5kdc[25035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:49:17.242516 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:44449
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:44449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:49:17.351300 25048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:17.351557 25048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:17.351596 25048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:17.355590 25048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:49:17.355711 25048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:17.355733 25048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:17.355751 25048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:49:17.355801 25048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:49:17.361008 25048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:44449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:44449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:17.362080 25048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:17.363102 25048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:17.371491 25054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:17.371572 25056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:17.371564 25053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:17.371568 25048 server_base.cc:1034] running on GCE node
I20260619 06:49:17.372433 25048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:17.373476 25048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:17.374670 25048 hybrid_clock.cc:648] HybridClock initialized: now 1781851757374658 us; error 41 us; skew 500 ppm
Jun 19 06:49:17 dist-test-slave-9x3b krb5kdc[25035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:49:17.377503 25048 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:49:17.378408 25048 webserver.cc:460] Webserver started at http://127.14.116.254:43641/ using document root <none> and password file <none>
I20260619 06:49:17.378911 25048 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:17.378966 25048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:17.379120 25048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:17.381107 25048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "3867597b5727473ebef5ee54608e32d1"
format_stamp: "Formatted at 2026-06-19 06:49:17 on dist-test-slave-9x3b"
server_key: "fe74f1591260b6e9c272f9456c15302e"
server_key_iv: "b768f279a1145e36b4ee168df7f5f8aa"
server_key_version: "encryptionkey@0"
I20260619 06:49:17.381606 25048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "3867597b5727473ebef5ee54608e32d1"
format_stamp: "Formatted at 2026-06-19 06:49:17 on dist-test-slave-9x3b"
server_key: "fe74f1591260b6e9c272f9456c15302e"
server_key_iv: "b768f279a1145e36b4ee168df7f5f8aa"
server_key_version: "encryptionkey@0"
I20260619 06:49:17.385159 25048 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 06:49:17.387408 25063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:17.388676 25048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:49:17.388772 25048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "3867597b5727473ebef5ee54608e32d1"
format_stamp: "Formatted at 2026-06-19 06:49:17 on dist-test-slave-9x3b"
server_key: "fe74f1591260b6e9c272f9456c15302e"
server_key_iv: "b768f279a1145e36b4ee168df7f5f8aa"
server_key_version: "encryptionkey@0"
I20260619 06:49:17.388851 25048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:17.438971 25048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:17.441890 25048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:17.442044 25048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:17.448523 25048 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:44449
I20260619 06:49:17.448809 25115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:44449 every 8 connection(s)
I20260619 06:49:17.449491 25048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:49:17.452150 25116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:49:17.457899 25116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1: Bootstrap starting.
I20260619 06:49:17.459342 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 25048
I20260619 06:49:17.459467 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:49:17.459790 14803 external_mini_cluster.cc:1428] Setting key d45edb73384a9cc3e858d36f463f1a04
I20260619 06:49:17.460686 25116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1: Neither blocks nor log segments found. Creating new log.
I20260619 06:49:17.461515 25116 log.cc:824] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:49:17.463388 25116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1: No bootstrap required, opened a new log
I20260619 06:49:17.466791 25116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3867597b5727473ebef5ee54608e32d1" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44449 } }
I20260619 06:49:17.467079 25116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:49:17.467167 25116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3867597b5727473ebef5ee54608e32d1, State: Initialized, Role: FOLLOWER
I20260619 06:49:17.467584 25116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3867597b5727473ebef5ee54608e32d1" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44449 } }
I20260619 06:49:17.467792 25116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:49:17.467862 25116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:49:17.467960 25116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 06:49:17 dist-test-slave-9x3b krb5kdc[25035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:49:17.469044 25116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3867597b5727473ebef5ee54608e32d1" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44449 } }
I20260619 06:49:17.469408 25116 leader_election.cc:302] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3867597b5727473ebef5ee54608e32d1; no voters:
I20260619 06:49:17.469786 25116 leader_election.cc:288] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:49:17.469923 25121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:49:17.470152 25121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [term 1 LEADER]: Becoming Leader. State: Replica: 3867597b5727473ebef5ee54608e32d1, State: Running, Role: LEADER
I20260619 06:49:17.470484 25121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3867597b5727473ebef5ee54608e32d1" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44449 } }
I20260619 06:49:17.471091 25116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:49:17.471714 25121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3867597b5727473ebef5ee54608e32d1. Latest consensus state: current_term: 1 leader_uuid: "3867597b5727473ebef5ee54608e32d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3867597b5727473ebef5ee54608e32d1" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44449 } } }
I20260619 06:49:17.471832 25121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:49:17.472163 25122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3867597b5727473ebef5ee54608e32d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3867597b5727473ebef5ee54608e32d1" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44449 } } }
I20260619 06:49:17.472221 25127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:49:17.472250 25122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1 [sys.catalog]: This master's current role is: LEADER
I20260619 06:49:17.475023 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:17.460942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.14.116.254:44449)
0619 06:49:17.461362 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:17.461379 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:17.461420 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:49:17.462269 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:17.462299 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:17.462616 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:17.462951 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:17.463835 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.464779 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.465631 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.466099 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.466272 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:17.469211 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:17.469216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:17.469230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:49:17.469256 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:17.471376 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:17.471971 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:17.471980 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:17.471988 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:17.472131 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:17.472463 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:17.472466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:17.472468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:17.472945 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:17.473678 (+ 733us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:17.473818 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 06:49:17.474096 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
I20260619 06:49:17.475396 25127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:49:17.481046 25127 catalog_manager.cc:1347] Generated new cluster ID: f41c06ce47794948821a2837f49302a4
I20260619 06:49:17.481115 25127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:49:17.489045 25127 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:49:17.490058 25127 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:49:17.496472 25127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1: Generated new TSK 0
I20260619 06:49:17.497119 25127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:49:17.524376 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44449
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:49:17.633074 25142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:17.633368 25142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:17.633455 25142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:17.637427 25142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:17.637521 25142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:17.637610 25142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:49:17.642849 25142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:17.643905 25142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:17.644912 25142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:17.654342 25150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:17.654419 25147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:17.654343 25148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:17.654655 25142 server_base.cc:1034] running on GCE node
I20260619 06:49:17.655040 25142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:17.655745 25142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:17.656935 25142 hybrid_clock.cc:648] HybridClock initialized: now 1781851757656881 us; error 79 us; skew 500 ppm
Jun 19 06:49:17 dist-test-slave-9x3b krb5kdc[25035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:49:17.659734 25142 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:49:17.660660 25142 webserver.cc:460] Webserver started at http://127.14.116.193:39347/ using document root <none> and password file <none>
I20260619 06:49:17.661234 25142 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:17.661329 25142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:17.661532 25142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:17.663527 25142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "d16166c58e0e4335835961d0633371e3"
format_stamp: "Formatted at 2026-06-19 06:49:17 on dist-test-slave-9x3b"
server_key: "9d95973a21b8d5a8f018d98108a988be"
server_key_iv: "f6b632cb300bfabd0ecdb198e0c29c86"
server_key_version: "encryptionkey@0"
I20260619 06:49:17.664163 25142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "d16166c58e0e4335835961d0633371e3"
format_stamp: "Formatted at 2026-06-19 06:49:17 on dist-test-slave-9x3b"
server_key: "9d95973a21b8d5a8f018d98108a988be"
server_key_iv: "f6b632cb300bfabd0ecdb198e0c29c86"
server_key_version: "encryptionkey@0"
I20260619 06:49:17.667737 25142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:49:17.670069 25157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:17.671365 25142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:49:17.671546 25142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "d16166c58e0e4335835961d0633371e3"
format_stamp: "Formatted at 2026-06-19 06:49:17 on dist-test-slave-9x3b"
server_key: "9d95973a21b8d5a8f018d98108a988be"
server_key_iv: "f6b632cb300bfabd0ecdb198e0c29c86"
server_key_version: "encryptionkey@0"
I20260619 06:49:17.671689 25142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:17.681228 25142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:17.684357 25142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:17.684553 25142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:17.685107 25142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:49:17.685901 25142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:49:17.685973 25142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:17.686038 25142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:49:17.686086 25142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:17.695366 25142 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:35053
I20260619 06:49:17.695386 25270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:35053 every 8 connection(s)
I20260619 06:49:17.696235 25142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:49:17.700861 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 25142
I20260619 06:49:17.700994 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:49:17.701313 14803 external_mini_cluster.cc:1428] Setting key b7bfbd100b92ff82da32f3ab2283a294
Jun 19 06:49:17 dist-test-slave-9x3b krb5kdc[25035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:49:17.711123 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:17.697956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:50303 (local address 127.14.116.254:44449)
0619 06:49:17.698116 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:17.698122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:17.698973 (+ 851us) server_negotiation.cc:407] Connection header received
0619 06:49:17.700388 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:17.700391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:17.700436 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:17.700511 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:17.702012 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.702560 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.703781 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.704048 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.704165 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:17.707081 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:17.707087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:17.707091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:49:17.707121 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:17.708911 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:17.709429 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:17.709435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:17.709440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:17.709547 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:17.709897 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:17.709903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:17.709906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:17.710151 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:17.710273 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:17.710786 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 06:49:17.710937 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
I20260619 06:49:17.711992 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:17.698346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44449 (local address 127.14.116.193:50303)
0619 06:49:17.698818 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:17.698860 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:17.700186 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:17.700720 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:17.700741 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:17.701246 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:17.701858 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:17.701874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.702690 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.702696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:17.703584 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:17.703593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.704360 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.704367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:17.704605 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:17.705231 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:17.705257 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:17.706910 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:49:17.709027 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:49:17.709033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:49:17.709045 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:49:17.709320 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:49:17.709642 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:49:17.709644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:49:17.709645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:49:17.709784 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:49:17.710266 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:49:17.710271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:49:17.710528 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 06:49:17.710692 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 06:49:17.710946 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260619 06:49:17.713433 25271 heartbeater.cc:347] Connected to a master server at 127.14.116.254:44449
I20260619 06:49:17.713745 25271 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
I20260619 06:49:17.714326 25271 heartbeater.cc:511] Master 127.14.116.254:44449 requested a full tablet report, sending...
I20260619 06:49:17.716032 25080 ts_manager.cc:194] Registered new tserver with Master: d16166c58e0e4335835961d0633371e3 (127.14.116.193:35053)
I20260619 06:49:17.717386 25080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:50303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:49:17.728837 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44449
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:49:17.835966 25276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:17.836215 25276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:17.836256 25276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:17.840184 25276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:17.840238 25276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:17.840294 25276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:49:17.845374 25276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:17.846290 25276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:17.847198 25276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:17.856091 25281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:17.856119 25284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:17.856186 25276 server_base.cc:1034] running on GCE node
W20260619 06:49:17.856106 25282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:17.856737 25276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:17.857364 25276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:17.858563 25276 hybrid_clock.cc:648] HybridClock initialized: now 1781851757858554 us; error 31 us; skew 500 ppm
Jun 19 06:49:17 dist-test-slave-9x3b krb5kdc[25035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:49:17.861325 25276 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:49:17.862172 25276 webserver.cc:460] Webserver started at http://127.14.116.194:36761/ using document root <none> and password file <none>
I20260619 06:49:17.862710 25276 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:17.862807 25276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:17.863047 25276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:17.865082 25276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "b3f00b7ac5b54a9eb129a184a9bbb996"
format_stamp: "Formatted at 2026-06-19 06:49:17 on dist-test-slave-9x3b"
server_key: "33c106abdf3befe42c1de2f439dff1ea"
server_key_iv: "c6a06150ad64ce8ac653e666e8b477bb"
server_key_version: "encryptionkey@0"
I20260619 06:49:17.865646 25276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "b3f00b7ac5b54a9eb129a184a9bbb996"
format_stamp: "Formatted at 2026-06-19 06:49:17 on dist-test-slave-9x3b"
server_key: "33c106abdf3befe42c1de2f439dff1ea"
server_key_iv: "c6a06150ad64ce8ac653e666e8b477bb"
server_key_version: "encryptionkey@0"
I20260619 06:49:17.869151 25276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:49:17.871564 25291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:17.872938 25276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:49:17.873093 25276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "b3f00b7ac5b54a9eb129a184a9bbb996"
format_stamp: "Formatted at 2026-06-19 06:49:17 on dist-test-slave-9x3b"
server_key: "33c106abdf3befe42c1de2f439dff1ea"
server_key_iv: "c6a06150ad64ce8ac653e666e8b477bb"
server_key_version: "encryptionkey@0"
I20260619 06:49:17.873178 25276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:17.898550 25276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:17.901888 25276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:17.902035 25276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:17.902581 25276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:49:17.903349 25276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:49:17.903395 25276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:17.903434 25276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:49:17.903450 25276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:17.911767 25276 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:41403
I20260619 06:49:17.911875 25404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:41403 every 8 connection(s)
I20260619 06:49:17.912662 25276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:49:17.914790 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 25276
I20260619 06:49:17.914901 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:49:17.915225 14803 external_mini_cluster.cc:1428] Setting key 19eb2c81f511c5ce0637c8de13f5dbc0
Jun 19 06:49:17 dist-test-slave-9x3b krb5kdc[25035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851757, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:49:17.926869 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:17.914381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:36339 (local address 127.14.116.254:44449)
0619 06:49:17.914535 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:17.914539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:17.915452 (+ 913us) server_negotiation.cc:407] Connection header received
0619 06:49:17.917004 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:17.917007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:17.917061 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:17.917140 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:17.918443 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.918952 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.920205 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.920609 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.920722 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:17.923268 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:17.923275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:17.923280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:49:17.923310 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:17.924959 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:17.925404 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:17.925407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:17.925408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:17.925480 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:17.925873 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:17.925875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:17.925876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:17.926036 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:17.926111 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:17.926651 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 06:49:17.926743 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:49:17.927937 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:17.914788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44449 (local address 127.14.116.194:36339)
0619 06:49:17.915260 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:17.915335 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:17.916790 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:17.917300 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:17.917307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:17.917642 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:17.918267 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:17.918288 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.919093 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.919096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:17.920075 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:17.920083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:17.920762 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:17.920765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:17.920909 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:17.921513 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:17.921529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:17.923098 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:49:17.925069 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:49:17.925073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:49:17.925084 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:49:17.925316 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:49:17.925595 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:49:17.925597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:49:17.925599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:49:17.925783 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:49:17.926123 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:49:17.926128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:49:17.926390 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 06:49:17.926601 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 06:49:17.926910 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":143,"threads_started":1}
I20260619 06:49:17.929342 25405 heartbeater.cc:347] Connected to a master server at 127.14.116.254:44449
I20260619 06:49:17.929575 25405 heartbeater.cc:464] Registering TS with master...
I20260619 06:49:17.930039 25405 heartbeater.cc:511] Master 127.14.116.254:44449 requested a full tablet report, sending...
I20260619 06:49:17.931159 25080 ts_manager.cc:194] Registered new tserver with Master: b3f00b7ac5b54a9eb129a184a9bbb996 (127.14.116.194:41403)
I20260619 06:49:17.932061 25080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:36339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:49:17.942441 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44449
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:49:18.049569 25410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:49:18.049808 25410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:49:18.049849 25410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:49:18.053818 25410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:49:18.053867 25410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:49:18.053984 25410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:49:18.059072 25410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:44449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:49:18.060107 25410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:49:18.061196 25410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:49:18.070674 25415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:18.070894 25410 server_base.cc:1034] running on GCE node
W20260619 06:49:18.070683 25416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:49:18.070746 25418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:49:18.071446 25410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:49:18.072165 25410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:49:18.073360 25410 hybrid_clock.cc:648] HybridClock initialized: now 1781851758073340 us; error 33 us; skew 500 ppm
Jun 19 06:49:18 dist-test-slave-9x3b krb5kdc[25035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851758, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:49:18.076143 25410 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:49:18.077062 25410 webserver.cc:460] Webserver started at http://127.14.116.195:38253/ using document root <none> and password file <none>
I20260619 06:49:18.077575 25410 fs_manager.cc:359] Metadata directory not provided
I20260619 06:49:18.077631 25410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:49:18.077862 25410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:49:18.079850 25410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "59be59d857f7498cb1bdb4f766e4c965"
format_stamp: "Formatted at 2026-06-19 06:49:18 on dist-test-slave-9x3b"
server_key: "947bc1347502c597e897493eb913e46d"
server_key_iv: "738f993e2674006c1f149f3fb65b1939"
server_key_version: "encryptionkey@0"
I20260619 06:49:18.080350 25410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "59be59d857f7498cb1bdb4f766e4c965"
format_stamp: "Formatted at 2026-06-19 06:49:18 on dist-test-slave-9x3b"
server_key: "947bc1347502c597e897493eb913e46d"
server_key_iv: "738f993e2674006c1f149f3fb65b1939"
server_key_version: "encryptionkey@0"
I20260619 06:49:18.083811 25410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:49:18.085996 25425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:18.087182 25410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:49:18.087306 25410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "59be59d857f7498cb1bdb4f766e4c965"
format_stamp: "Formatted at 2026-06-19 06:49:18 on dist-test-slave-9x3b"
server_key: "947bc1347502c597e897493eb913e46d"
server_key_iv: "738f993e2674006c1f149f3fb65b1939"
server_key_version: "encryptionkey@0"
I20260619 06:49:18.087387 25410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:49:18.102425 25410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:49:18.105225 25410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:49:18.105381 25410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:49:18.105873 25410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:49:18.106595 25410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:49:18.106640 25410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:18.106678 25410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:49:18.106693 25410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:49:18.115144 25410 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:46591
I20260619 06:49:18.115165 25538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:46591 every 8 connection(s)
I20260619 06:49:18.116070 25410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:49:18.118460 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 25410
I20260619 06:49:18.118598 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:49:18.118928 14803 external_mini_cluster.cc:1428] Setting key be51eb1e5f28efbdc2bd63149339ce47
Jun 19 06:49:18 dist-test-slave-9x3b krb5kdc[25035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851758, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:49:18.129128 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.117857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:37017 (local address 127.14.116.254:44449)
0619 06:49:18.117990 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:18.117994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:18.118841 (+ 847us) server_negotiation.cc:407] Connection header received
0619 06:49:18.120524 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:18.120527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:18.120578 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:18.120664 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:18.121852 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.122371 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.123435 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.123727 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.123818 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.126110 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:18.126113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:18.126115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:49:18.126136 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:18.127413 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:18.127860 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:18.127863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:18.127864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:18.127936 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:18.128203 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:18.128205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:18.128206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:18.128383 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:18.128456 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:18.128908 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 06:49:18.129016 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:49:18.129920 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.118225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:44449 (local address 127.14.116.195:37017)
0619 06:49:18.118675 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:18.118727 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:18.120334 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:18.120818 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:18.120824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:18.121262 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:49:18.121727 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.121740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.122474 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.122476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.123315 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.123322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.123857 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.123860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.123996 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.124533 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:49:18.124548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:49:18.125967 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:49:18.127508 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:49:18.127512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:49:18.127522 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:49:18.127775 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:49:18.128015 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:49:18.128017 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:49:18.128018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:49:18.128134 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:49:18.128467 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:49:18.128472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:49:18.128704 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 06:49:18.128858 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 06:49:18.129099 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":137,"threads_started":1}
I20260619 06:49:18.131047 25539 heartbeater.cc:347] Connected to a master server at 127.14.116.254:44449
I20260619 06:49:18.131287 25539 heartbeater.cc:464] Registering TS with master...
I20260619 06:49:18.131915 25539 heartbeater.cc:511] Master 127.14.116.254:44449 requested a full tablet report, sending...
I20260619 06:49:18.132956 25080 ts_manager.cc:194] Registered new tserver with Master: 59be59d857f7498cb1bdb4f766e4c965 (127.14.116.195:46591)
I20260619 06:49:18.133494 25080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:37017
I20260619 06:49:18.144471 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:49:18.153443 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.146277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.14.116.254:44449)
0619 06:49:18.146471 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:18.146476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:18.146618 (+ 142us) server_negotiation.cc:407] Connection header received
0619 06:49:18.146771 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:18.146774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:18.146825 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:18.146916 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:49:18.147578 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.148134 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.148971 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.149271 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.149390 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.150589 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:49:18.150592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:49:18.150595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:49:18.150619 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:49:18.151820 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:18.152214 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:18.152219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:18.152221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:18.152301 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:49:18.152569 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:49:18.152574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:49:18.152576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:49:18.152782 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:49:18.152927 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:18.153151 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 06:49:18.153271 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 06:49:18.155967 25080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:49:18.158811 25080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:49:18.172127 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.166515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:41403 (local address 127.0.0.1:46096)
0619 06:49:18.166849 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:18.166898 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:18.167022 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:18.168222 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:18.168228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:18.168249 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:49:18.168394 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.168402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.169483 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.169489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.170682 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.170691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.171516 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.171547 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.171712 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.171760 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 06:49:18.171878 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 06:49:18.171989 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260619 06:49:18.172196 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.166687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.14.116.194:41403)
0619 06:49:18.167804 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:18.167816 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:18.167839 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:49:18.167917 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:18.167927 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:18.168069 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:18.168235 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:49:18.168510 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.169350 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.170829 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.171378 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.171537 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.171725 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:18.171952 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 06:49:18.172045 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":87,"threads_started":1}
I20260619 06:49:18.172889 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.166863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:35053 (local address 127.0.0.1:33340)
0619 06:49:18.167243 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:18.167256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:18.167334 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:18.168233 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:18.168238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:18.168276 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:49:18.168407 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.168414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.169983 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.169987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.171336 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.171346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.172435 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.172438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.172520 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.172541 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:49:18.172667 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 06:49:18.172740 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":58,"spinlock_wait_cycles":11904,"thread_start_us":93,"threads_started":1}
I20260619 06:49:18.172911 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.167288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.14.116.193:35053)
0619 06:49:18.167760 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:18.167771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:18.167795 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:49:18.167875 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:18.167880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:18.168079 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:18.168548 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:49:18.168693 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.169833 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.171467 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.172329 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.172449 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.172561 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:18.172681 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 06:49:18.172762 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":124,"threads_started":1}
I20260619 06:49:18.173043 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.167223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:46591 (local address 127.0.0.1:51416)
0619 06:49:18.167560 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:18.167574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:18.167678 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:18.168358 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:18.168360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:18.168373 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:49:18.168448 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.168455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.169983 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.169987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.171342 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.171347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.172643 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.172647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.172744 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.172769 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:49:18.172818 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 06:49:18.172890 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":136,"thread_start_us":82,"threads_started":1}
I20260619 06:49:18.174356 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.167587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.14.116.195:46591)
0619 06:49:18.167968 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:18.167978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:18.167996 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:49:18.168060 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:18.168064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:18.168221 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:18.168568 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:49:18.168703 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.169833 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.171671 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.172488 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.173988 (+ 1500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.174073 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:18.174109 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 06:49:18.174169 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
I20260619 06:49:18.175730 25339 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6a94af4fb7405eac1bd06da70a67f4 (DEFAULT_TABLE table=test-table [id=356be9e590ef43829e369d772b3dee1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:49:18.175730 25205 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6a94af4fb7405eac1bd06da70a67f4 (DEFAULT_TABLE table=test-table [id=356be9e590ef43829e369d772b3dee1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:49:18.176839 25205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6a94af4fb7405eac1bd06da70a67f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:49:18.176839 25339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6a94af4fb7405eac1bd06da70a67f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:49:18.178118 25473 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6a94af4fb7405eac1bd06da70a67f4 (DEFAULT_TABLE table=test-table [id=356be9e590ef43829e369d772b3dee1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:49:18.178861 25473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6a94af4fb7405eac1bd06da70a67f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:49:18.183977 25556 tablet_bootstrap.cc:492] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3: Bootstrap starting.
I20260619 06:49:18.184741 25558 tablet_bootstrap.cc:492] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: Bootstrap starting.
I20260619 06:49:18.186126 25556 tablet_bootstrap.cc:654] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3: Neither blocks nor log segments found. Creating new log.
I20260619 06:49:18.186569 25558 tablet_bootstrap.cc:654] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: Neither blocks nor log segments found. Creating new log.
I20260619 06:49:18.187367 25556 log.cc:824] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:49:18.187731 25558 log.cc:824] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: Log is configured to *not* fsync() on all Append() calls
I20260619 06:49:18.188100 25557 tablet_bootstrap.cc:492] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: Bootstrap starting.
I20260619 06:49:18.189466 25558 tablet_bootstrap.cc:492] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: No bootstrap required, opened a new log
I20260619 06:49:18.189700 25558 ts_tablet_manager.cc:1397] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:49:18.190251 25557 tablet_bootstrap.cc:654] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: Neither blocks nor log segments found. Creating new log.
I20260619 06:49:18.190979 25557 log.cc:824] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: Log is configured to *not* fsync() on all Append() calls
I20260619 06:49:18.192546 25556 tablet_bootstrap.cc:492] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3: No bootstrap required, opened a new log
I20260619 06:49:18.192790 25556 ts_tablet_manager.cc:1397] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 06:49:18.193102 25557 tablet_bootstrap.cc:492] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: No bootstrap required, opened a new log
I20260619 06:49:18.193302 25557 ts_tablet_manager.cc:1397] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:49:18.193295 25558 raft_consensus.cc:348] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } }
I20260619 06:49:18.193562 25558 raft_consensus.cc:374] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:49:18.193642 25558 raft_consensus.cc:729] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59be59d857f7498cb1bdb4f766e4c965, State: Initialized, Role: FOLLOWER
I20260619 06:49:18.194087 25558 consensus_queue.cc:260] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } }
I20260619 06:49:18.195142 25558 ts_tablet_manager.cc:1428] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:49:18.195500 25539 heartbeater.cc:503] Master 127.14.116.254:44449 was elected leader, sending a full tablet report...
I20260619 06:49:18.196591 25557 raft_consensus.cc:348] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } }
I20260619 06:49:18.196628 25556 raft_consensus.cc:348] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } }
I20260619 06:49:18.196815 25557 raft_consensus.cc:374] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:49:18.196872 25557 raft_consensus.cc:729] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f00b7ac5b54a9eb129a184a9bbb996, State: Initialized, Role: FOLLOWER
I20260619 06:49:18.196909 25556 raft_consensus.cc:374] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:49:18.196959 25556 raft_consensus.cc:729] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d16166c58e0e4335835961d0633371e3, State: Initialized, Role: FOLLOWER
I20260619 06:49:18.197314 25557 consensus_queue.cc:260] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } }
I20260619 06:49:18.197381 25556 consensus_queue.cc:260] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } }
I20260619 06:49:18.197942 25405 heartbeater.cc:503] Master 127.14.116.254:44449 was elected leader, sending a full tablet report...
I20260619 06:49:18.198236 25557 ts_tablet_manager.cc:1428] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:49:18.198340 25556 ts_tablet_manager.cc:1428] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 06:49:18.198403 25272 tablet.cc:2369] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:49:18.198663 25271 heartbeater.cc:503] Master 127.14.116.254:44449 was elected leader, sending a full tablet report...
I20260619 06:49:18.288403 25562 raft_consensus.cc:482] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:49:18.288607 25562 raft_consensus.cc:504] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } }
I20260619 06:49:18.289755 25562 leader_election.cc:288] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3f00b7ac5b54a9eb129a184a9bbb996 (127.14.116.194:41403), d16166c58e0e4335835961d0633371e3 (127.14.116.193:35053)
I20260619 06:49:18.293895 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.290118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:49177 (local address 127.14.116.193:35053)
0619 06:49:18.290235 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:18.290242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:18.290533 (+ 291us) server_negotiation.cc:407] Connection header received
0619 06:49:18.290677 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:18.290686 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:18.290725 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:18.290804 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:49:18.291073 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.291697 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.292888 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.293348 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.293473 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.293527 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:18.293699 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 06:49:18.293764 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:49:18.293917 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.289986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:41403 (local address 127.14.116.195:46375)
0619 06:49:18.290106 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:18.290121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:18.290207 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:18.290575 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:18.290578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:18.290595 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:49:18.290711 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.290717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.291706 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.291712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.292744 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.292752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.293476 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.293480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.293598 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.293615 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 06:49:18.293704 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 06:49:18.293774 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":64}
I20260619 06:49:18.293901 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.290190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:46375 (local address 127.14.116.194:41403)
0619 06:49:18.290327 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:18.290332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:18.290348 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:49:18.290417 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:18.290419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:18.290459 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:18.290552 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:49:18.290890 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.291570 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.292866 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.293350 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.293476 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.293528 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:18.293718 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 06:49:18.293780 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:49:18.293934 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.289994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:35053 (local address 127.14.116.195:49177)
0619 06:49:18.290437 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:18.290448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:49:18.290561 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:49:18.290825 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:49:18.290828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:49:18.290850 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:49:18.290961 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.290966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.291800 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.291804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.292764 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:49:18.292773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.293465 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.293468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:49:18.293565 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.293589 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:49:18.293685 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:49:18.293752 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":154,"thread_start_us":114,"threads_started":1}
I20260619 06:49:18.294605 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6a94af4fb7405eac1bd06da70a67f4" candidate_uuid: "59be59d857f7498cb1bdb4f766e4c965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" is_pre_election: true
I20260619 06:49:18.294692 25225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6a94af4fb7405eac1bd06da70a67f4" candidate_uuid: "59be59d857f7498cb1bdb4f766e4c965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16166c58e0e4335835961d0633371e3" is_pre_election: true
I20260619 06:49:18.294950 25359 raft_consensus.cc:2454] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59be59d857f7498cb1bdb4f766e4c965 in term 0.
I20260619 06:49:18.294976 25225 raft_consensus.cc:2454] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59be59d857f7498cb1bdb4f766e4c965 in term 0.
I20260619 06:49:18.295418 25426 leader_election.cc:302] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 59be59d857f7498cb1bdb4f766e4c965, b3f00b7ac5b54a9eb129a184a9bbb996, d16166c58e0e4335835961d0633371e3; no voters:
I20260619 06:49:18.295706 25562 raft_consensus.cc:2781] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:49:18.295805 25562 raft_consensus.cc:482] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:49:18.295866 25562 raft_consensus.cc:3037] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:49:18.296859 25562 raft_consensus.cc:504] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } }
I20260619 06:49:18.297223 25562 leader_election.cc:288] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [CANDIDATE]: Term 1 election: Requested vote from peers b3f00b7ac5b54a9eb129a184a9bbb996 (127.14.116.194:41403), d16166c58e0e4335835961d0633371e3 (127.14.116.193:35053)
I20260619 06:49:18.297561 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6a94af4fb7405eac1bd06da70a67f4" candidate_uuid: "59be59d857f7498cb1bdb4f766e4c965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996"
I20260619 06:49:18.297595 25225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6a94af4fb7405eac1bd06da70a67f4" candidate_uuid: "59be59d857f7498cb1bdb4f766e4c965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16166c58e0e4335835961d0633371e3"
I20260619 06:49:18.297686 25359 raft_consensus.cc:3037] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:49:18.297727 25225 raft_consensus.cc:3037] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:49:18.298702 25225 raft_consensus.cc:2454] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59be59d857f7498cb1bdb4f766e4c965 in term 1.
I20260619 06:49:18.298888 25359 raft_consensus.cc:2454] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59be59d857f7498cb1bdb4f766e4c965 in term 1.
I20260619 06:49:18.298985 25429 leader_election.cc:302] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59be59d857f7498cb1bdb4f766e4c965, d16166c58e0e4335835961d0633371e3; no voters:
I20260619 06:49:18.299129 25562 raft_consensus.cc:2781] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:49:18.299332 25562 raft_consensus.cc:686] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 1 LEADER]: Becoming Leader. State: Replica: 59be59d857f7498cb1bdb4f766e4c965, State: Running, Role: LEADER
I20260619 06:49:18.299698 25562 consensus_queue.cc:237] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } }
I20260619 06:49:18.302661 25080 catalog_manager.cc:5495] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59be59d857f7498cb1bdb4f766e4c965 (127.14.116.195). New cstate: current_term: 1 leader_uuid: "59be59d857f7498cb1bdb4f766e4c965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59be59d857f7498cb1bdb4f766e4c965" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 46591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 } health_report { overall_health: UNKNOWN } } }
I20260619 06:49:18.321415 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.318174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.14.116.195:46591)
0619 06:49:18.318296 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:49:18.318300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:49:18.318312 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:49:18.318450 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:49:18.318453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:49:18.318506 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:49:18.318592 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:49:18.318795 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.319268 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.320122 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:49:18.320322 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:49:18.320405 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:49:18.320633 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:49:18.321066 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:49:18.321154 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:49:18.321205 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 06:49:18.321252 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:49:18.326881 25359 raft_consensus.cc:1261] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [term 1 FOLLOWER]: Refusing update from remote peer 59be59d857f7498cb1bdb4f766e4c965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:49:18.326912 25225 raft_consensus.cc:1261] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3 [term 1 FOLLOWER]: Refusing update from remote peer 59be59d857f7498cb1bdb4f766e4c965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:49:18.327570 25566 consensus_queue.cc:1035] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f00b7ac5b54a9eb129a184a9bbb996" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:49:18.327777 25562 consensus_queue.cc:1035] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d16166c58e0e4335835961d0633371e3" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:49:18.335649 25569 mvcc.cc:205] Tried to move back new op lower bound from 7298464802100531200 to 7298464801998196736. Current Snapshot: MvccSnapshot[applied={T|T < 7298464802100531200 or (T in {7298464802100531200})}]
I20260619 06:49:18.342535 25079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:49:18.342707 25079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:49:18.345091 25079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3867597b5727473ebef5ee54608e32d1: Sending DeleteTablet for 3 replicas of tablet 3a6a94af4fb7405eac1bd06da70a67f4
I20260619 06:49:18.345757 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6a94af4fb7405eac1bd06da70a67f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:49:18 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:46096
I20260619 06:49:18.345857 25473 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6a94af4fb7405eac1bd06da70a67f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:49:18 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:51416
I20260619 06:49:18.346243 25579 tablet_replica.cc:330] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: stopping tablet replica
I20260619 06:49:18.346242 25205 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6a94af4fb7405eac1bd06da70a67f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:49:18 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:33340
I20260619 06:49:18.346549 25579 raft_consensus.cc:2229] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:49:18.346846 25581 tablet_replica.cc:330] T 3a6a94af4fb7405eac1bd06da70a67f4 P d16166c58e0e4335835961d0633371e3: stopping tablet replica
I20260619 06:49:18.346911 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 25142
I20260619 06:49:18.347867 25579 raft_consensus.cc:2258] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:49:18.348110 25580 tablet_replica.cc:330] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: stopping tablet replica
I20260619 06:49:18.348368 25580 raft_consensus.cc:2229] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:49:18.348692 25580 raft_consensus.cc:2258] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:49:18.349732 25579 ts_tablet_manager.cc:1905] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:49:18.350172 25580 ts_tablet_manager.cc:1905] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:49:18.352845 25580 ts_tablet_manager.cc:1918] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:49:18.353020 25580 log.cc:1196] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/3a6a94af4fb7405eac1bd06da70a67f4
I20260619 06:49:18.353189 25579 ts_tablet_manager.cc:1918] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:49:18.353380 25579 log.cc:1196] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/3a6a94af4fb7405eac1bd06da70a67f4
I20260619 06:49:18.353430 25580 ts_tablet_manager.cc:1939] T 3a6a94af4fb7405eac1bd06da70a67f4 P 59be59d857f7498cb1bdb4f766e4c965: Deleting consensus metadata
I20260619 06:49:18.353772 25579 ts_tablet_manager.cc:1939] T 3a6a94af4fb7405eac1bd06da70a67f4 P b3f00b7ac5b54a9eb129a184a9bbb996: Deleting consensus metadata
I20260619 06:49:18.354657 25064 catalog_manager.cc:4841] TS 59be59d857f7498cb1bdb4f766e4c965 (127.14.116.195:46591): tablet 3a6a94af4fb7405eac1bd06da70a67f4 (table test-table [id=356be9e590ef43829e369d772b3dee1f]) successfully deleted
W20260619 06:49:18.354986 25067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:35053 (error 108)
I20260619 06:49:18.355089 25064 catalog_manager.cc:4841] TS b3f00b7ac5b54a9eb129a184a9bbb996 (127.14.116.194:41403): tablet 3a6a94af4fb7405eac1bd06da70a67f4 (table test-table [id=356be9e590ef43829e369d772b3dee1f]) successfully deleted
I20260619 06:49:18.355737 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 25276
I20260619 06:49:18.356145 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:49:18.355783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:35053 (local address 127.0.0.1:33344)
0619 06:49:18.355987 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 06:49:18.356054 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260619 06:49:18.356566 25067 catalog_manager.cc:4584] TS d16166c58e0e4335835961d0633371e3 (127.14.116.193:35053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a6a94af4fb7405eac1bd06da70a67f4: Network error: Client connection negotiation failed: client connection to 127.14.116.193:35053: connect: Connection refused (error 111)
I20260619 06:49:18.362905 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 25410
I20260619 06:49:18.367605 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 25048
2026-06-19T06:49:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2284 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:49:18 dist-test-slave-9x3b krb5kdc[25585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:49:18 dist-test-slave-9x3b krb5kdc[25585](info): set up 2 sockets
Jun 19 06:49:18 dist-test-slave-9x3b krb5kdc[25585](info): commencing operation
krb5kdc: starting...
W20260619 06:49:19.408601 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:49:19 dist-test-slave-9x3b krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:49:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/rangeradmin_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/rangeradmin_127.14.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/rangerlookup_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/rangerlookup_127.14.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/HTTP_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/HTTP_127.14.116.212@KRBTEST.COM.keytab.
I20260619 06:49:19.564209 14803 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2eB_D1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/postgres -l logfile start
2026-06-19 06:49:21.904 UTC [25615] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:49:21.905 UTC [25615] LOG: listening on IPv4 address "127.14.116.212", port 54645
2026-06-19 06:49:21.908 UTC [25615] LOG: listening on Unix socket "/tmp/.s.PGSQL.54645"
2026-06-19 06:49:21.928 UTC [25618] LOG: database system was shut down at 2026-06-19 06:49:20 UTC
2026-06-19 06:49:21.932 UTC [25615] LOG: database system is ready to accept connections
I20260619 06:49:22.876325 14803 mini_postgres.cc:96] Postgres bound to 54645
2026-06-19 06:49:22.882 UTC [25626] FATAL: database "slave" does not exist
127.14.116.212:54645 - accepting connections
I20260619 06:49:22.883103 14803 mini_ranger.cc:161] Starting Ranger...
I20260619 06:49:22.898304 14803 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:49:23.327303 14803 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:49:23.327401 14803 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-admin
2026-06-19 06:49:23,602 [I] DB FLAVOR :POSTGRES
2026-06-19 06:49:23,602 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:49:23,602 [I] Checking connection..
2026-06-19 06:49:23,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:49:23,844 [I] Checking connection passed.
2026-06-19 06:49:23,845 [I] --------- Verifying version history table ---------
2026-06-19 06:49:23,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:49:24,097 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:49:24,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:49:24,341 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:49:24,342 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:49:24,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:49:24.568 UTC [25690] WARNING: there is no transaction in progress
2026-06-19 06:49:24,574 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:49:24,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:49:24,804 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:49:24,804 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:49:24,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:49:25,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:49:25,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9x3b.c.gcp-upstream.internal','N') ;"
2026-06-19 06:49:25,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:49:25,726 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:49:25,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:49:25,982 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:49:25,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:49:26,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:49:26,459 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:49:26,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:49:27.371 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.387 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.403 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.417 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.432 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.450 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.500 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.507 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.520 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.529 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.539 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.552 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.561 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.567 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.573 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:27.579 UTC [25798] WARNING: there is no transaction in progress
2026-06-19 06:49:28,165 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:49:28,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:49:28,383 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:49:28,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:49:28,638 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:49:28,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:49:28,890 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:49:28,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:49:29,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:49:29,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9x3b.c.gcp-upstream.internal';"
2026-06-19 06:49:29,599 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:49:29,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:49:29,826 [I] DB_PATCHES have already been applied
I20260619 06:49:29.832140 14803 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-admin:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2eB_D1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:49:29.832221 14803 mini_ranger.cc:193] Using host: 127.14.116.212
I20260619 06:49:29.835417 14803 mini_ranger.cc:227] Ranger admin URL: http://127.14.116.212:56089
Jun 19, 2026 6:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:49:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:49:30.942555 14803 mini_ranger.cc:160] Time spent starting Ranger: real 8.059s user 0.000s sys 0.010s
Jun 19, 2026 6:49:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:49:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:49:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:49:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:49:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:49:51.658895 14803 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/rangerkms_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/rangerkms_127.14.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.116.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.116.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.116.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/HTTP_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.116.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/HTTP_127.14.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:49:51 dist-test-slave-9x3b krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851791, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:49:51.741977 14803 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:49:51.751838 14803 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:49:52.526160 14803 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:49:52.526257 14803 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-kms
I20260619 06:49:52.526376 14803 mini_ranger_kms.cc:227] Using postgres at 127.14.116.212:54645
2026-06-19 06:49:52,710 [I] DB FLAVOR :POSTGRES
2026-06-19 06:49:52,710 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:49:52,710 [I] Checking connection
2026-06-19 06:49:52,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:49:52,944 [I] connection success
2026-06-19 06:49:52,944 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:49:52,944 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:49:52,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:49:53,179 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:49:53,180 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:49:53,180 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:49:53,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:54645/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:49:53,450 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:49:53.924233 14803 mini_ranger_kms.cc:310] Created kms service
I20260619 06:49:54.116966 14803 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:49:54.158331 14803 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:49:54 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851791, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:49:54.536330 14803 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:49:54.536471 14803 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-kms:/tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:49:54.536501 14803 mini_ranger_kms.cc:240] Using host: 127.14.116.212
I20260619 06:49:54.540335 14803 mini_ranger_kms.cc:276] Ranger KMS PID: 25978
I20260619 06:49:54.540452 14803 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.116.212:49045
Jun 19, 2026 6:49:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:49:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:49:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:49:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:49:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:49:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 06:49:56.739154 14803 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.997s user 0.013s sys 0.156s
I20260619 06:49:56.739279 14803 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:49:56.739401 14803 mini_ranger_kms.cc:401] 127.14.116.212:49045/kms/v1/keys
Jun 19 06:49:56 dist-test-slave-9x3b krb5kdc[25585](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851796, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 06:49:57 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851796, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.212@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
Jun 19 06:49:57 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851796, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.212@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
Jun 19 06:49:59 dist-test-slave-9x3b krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:50:00.013024 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:39421
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:39421
--ranger_config_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:49045/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:00.130538 26029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:00.130996 26029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:00.131114 26029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:00.138465 26029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:00.138573 26029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:00.138607 26029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:00.138657 26029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:00.138713 26029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:00.146611 26029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.116.212:49045/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:39421
--ranger_config_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:00.148243 26029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:00.149657 26029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:00.159965 26035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:00.159955 26034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:00.160392 26037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:00.160815 26029 server_base.cc:1034] running on GCE node
I20260619 06:50:00.161517 26029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:00.162678 26029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:00.163848 26029 hybrid_clock.cc:648] HybridClock initialized: now 1781851800163776 us; error 88 us; skew 500 ppm
Jun 19 06:50:00 dist-test-slave-9x3b krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851800, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:00.167469 26029 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:50:00.168540 26029 webserver.cc:460] Webserver started at http://127.14.116.254:43383/ using document root <none> and password file <none>
I20260619 06:50:00.169181 26029 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:00.169256 26029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:00.169649 26029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:50:00 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851800, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:00.222018 26029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "8b442dd9618d412cbe2672e8a24c1ca6"
format_stamp: "Formatted at 2026-06-19 06:50:00 on dist-test-slave-9x3b"
server_key: "137f32f60209da3bef3bfed1b5234948"
server_key_iv: "261b3638060aadd949e4a0375a2d2379"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:00.222766 26029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "8b442dd9618d412cbe2672e8a24c1ca6"
format_stamp: "Formatted at 2026-06-19 06:50:00 on dist-test-slave-9x3b"
server_key: "137f32f60209da3bef3bfed1b5234948"
server_key_iv: "261b3638060aadd949e4a0375a2d2379"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:00.228103 26029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:50:00.245957 26044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:00.247674 26029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:50:00.247814 26029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "8b442dd9618d412cbe2672e8a24c1ca6"
format_stamp: "Formatted at 2026-06-19 06:50:00 on dist-test-slave-9x3b"
server_key: "137f32f60209da3bef3bfed1b5234948"
server_key_iv: "261b3638060aadd949e4a0375a2d2379"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:00.247965 26029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:00.267118 26029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:00.270134 26029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:00.270340 26029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:00.278386 26029 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:39421
I20260619 06:50:00.278429 26096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:39421 every 8 connection(s)
I20260619 06:50:00.279390 26029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:50:00.281330 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 26029
I20260619 06:50:00.281610 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:50:00.281929 14803 external_mini_cluster.cc:1428] Setting key 395518dc2823f011c511d4fb9f096362
I20260619 06:50:00.283181 26097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:00.290230 26097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6: Bootstrap starting.
Jun 19 06:50:00 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:00.292922 26097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:00.294281 26097 log.cc:824] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:00.296627 26097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6: No bootstrap required, opened a new log
I20260619 06:50:00.296732 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:00.283358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.14.116.254:39421)
0619 06:50:00.283801 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:00.283831 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:00.284033 (+ 202us) server_negotiation.cc:407] Connection header received
0619 06:50:00.284800 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:00.284816 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:00.285148 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:00.285486 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:00.286262 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:00.287367 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:00.288364 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:00.288839 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:00.289013 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:00.291651 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:00.291659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:00.291676 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:00.291710 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:00.293438 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:00.293907 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:00.293912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:00.293917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:00.294018 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:00.294347 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:00.294350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:00.294352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:00.294858 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:00.295069 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:00.295375 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 06:50:00.295838 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":147,"threads_started":1}
I20260619 06:50:00.300832 26097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b442dd9618d412cbe2672e8a24c1ca6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39421 } }
I20260619 06:50:00.301044 26097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:00.301079 26097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b442dd9618d412cbe2672e8a24c1ca6, State: Initialized, Role: FOLLOWER
I20260619 06:50:00.301556 26097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b442dd9618d412cbe2672e8a24c1ca6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39421 } }
I20260619 06:50:00.301705 26097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:50:00.301761 26097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:50:00.301848 26097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:00.302831 26097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b442dd9618d412cbe2672e8a24c1ca6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39421 } }
I20260619 06:50:00.303169 26097 leader_election.cc:302] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b442dd9618d412cbe2672e8a24c1ca6; no voters:
I20260619 06:50:00.303584 26097 leader_election.cc:288] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:50:00.303715 26102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:50:00.304131 26102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 8b442dd9618d412cbe2672e8a24c1ca6, State: Running, Role: LEADER
I20260619 06:50:00.304445 26102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b442dd9618d412cbe2672e8a24c1ca6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39421 } }
I20260619 06:50:00.304890 26097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:50:00.305629 26102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b442dd9618d412cbe2672e8a24c1ca6. Latest consensus state: current_term: 1 leader_uuid: "8b442dd9618d412cbe2672e8a24c1ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b442dd9618d412cbe2672e8a24c1ca6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39421 } } }
I20260619 06:50:00.305738 26102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:00.306077 26111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:50:00.306255 26104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b442dd9618d412cbe2672e8a24c1ca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b442dd9618d412cbe2672e8a24c1ca6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 39421 } } }
I20260619 06:50:00.306422 26104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:00.309405 26111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:50:00.309789 26097 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:50:00.326138 26111 catalog_manager.cc:1347] Generated new cluster ID: a3c8e59f54394506bad471cb629dd77c
I20260619 06:50:00.326218 26111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:50:00.341596 26111 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:50:00.342986 26111 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:50:00.354704 26111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6: Generated new TSK 0
I20260619 06:50:00.356325 26111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:50:01.046442 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:49045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39421
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:01.169698 26142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:01.169992 26142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:01.170081 26142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:01.174424 26142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:01.174556 26142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:01.174690 26142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:50:01.180531 26142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.116.212:49045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:01.181617 26142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:01.182708 26142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:01.193719 26150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:01.193985 26148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:01.193926 26142 server_base.cc:1034] running on GCE node
W20260619 06:50:01.193709 26147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:01.194495 26142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:01.195118 26142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:01.196298 26142 hybrid_clock.cc:648] HybridClock initialized: now 1781851801196278 us; error 39 us; skew 500 ppm
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:01.199702 26142 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:50:01.200824 26142 webserver.cc:460] Webserver started at http://127.14.116.193:39029/ using document root <none> and password file <none>
I20260619 06:50:01.201478 26142 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:01.201570 26142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:01.201828 26142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:01.221825 26142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "9c2d67132c6f4b559a5f8a0bd0cce115"
format_stamp: "Formatted at 2026-06-19 06:50:01 on dist-test-slave-9x3b"
server_key: "cabaae4a659b5237273a93c55a261ff7"
server_key_iv: "d34ae23e098f37fba60348149ca51812"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:01.222543 26142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "9c2d67132c6f4b559a5f8a0bd0cce115"
format_stamp: "Formatted at 2026-06-19 06:50:01 on dist-test-slave-9x3b"
server_key: "cabaae4a659b5237273a93c55a261ff7"
server_key_iv: "d34ae23e098f37fba60348149ca51812"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:01.227689 26142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 06:50:01.248971 26159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:01.250821 26142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:50:01.251030 26142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "9c2d67132c6f4b559a5f8a0bd0cce115"
format_stamp: "Formatted at 2026-06-19 06:50:01 on dist-test-slave-9x3b"
server_key: "cabaae4a659b5237273a93c55a261ff7"
server_key_iv: "d34ae23e098f37fba60348149ca51812"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:01.251255 26142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:01.271211 26142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:01.274945 26142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:01.275139 26142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:01.275877 26142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:01.276904 26142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:01.276973 26142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:01.277026 26142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:01.277055 26142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:01.287737 26142 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:45205
I20260619 06:50:01.288868 26142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:50:01.298031 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 26142
I20260619 06:50:01.298170 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:50:01.298484 14803 external_mini_cluster.cc:1428] Setting key e09084604fb1781d0d10b9ef700c35dd
I20260619 06:50:01.304445 26272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:45205 every 8 connection(s)
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:01.336282 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:01.306238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:35725 (local address 127.14.116.254:39421)
0619 06:50:01.317556 (+ 11318us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:01.317565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:01.317580 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:50:01.317621 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:01.317623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:01.317671 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:01.317755 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:01.324909 (+ 7154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:01.325759 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:01.327526 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:01.327969 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:01.328130 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:01.331539 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:01.331543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:01.331546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:01.331586 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:01.333538 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:01.334270 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:01.334274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:01.334276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:01.334376 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:01.334832 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:01.334839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:01.334844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:01.335104 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:01.335187 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:01.335955 (+ 768us) server_negotiation.cc:299] Negotiation successful
0619 06:50:01.336089 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11235,"thread_start_us":117,"threads_started":1}
I20260619 06:50:01.337375 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:01.306772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:39421 (local address 127.14.116.193:35725)
0619 06:50:01.315954 (+ 9182us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:01.316025 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:01.317106 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:01.323922 (+ 6816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:01.323930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:01.324208 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:01.324719 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:01.324733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:01.325936 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:01.325940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:01.327333 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:01.327346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:01.328358 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:01.328364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:01.328584 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:01.329299 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:01.329332 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:01.331308 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:01.333682 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:01.333720 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:01.333741 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:01.334136 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:01.334491 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:01.334495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:01.334496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:01.334692 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:01.335271 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:01.335281 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:01.335653 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 06:50:01.335882 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 06:50:01.336248 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8922,"thread_start_us":130,"threads_started":1}
I20260619 06:50:01.338728 26273 heartbeater.cc:347] Connected to a master server at 127.14.116.254:39421
I20260619 06:50:01.338973 26273 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:01.339524 26273 heartbeater.cc:511] Master 127.14.116.254:39421 requested a full tablet report, sending...
I20260619 06:50:01.341265 26057 ts_manager.cc:194] Registered new tserver with Master: 9c2d67132c6f4b559a5f8a0bd0cce115 (127.14.116.193:45205)
I20260619 06:50:01.342767 26057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:35725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:50:01.351218 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:49045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39421
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:50:01.506547 26281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:01.506824 26281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:01.506916 26281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:01.511350 26281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:01.511445 26281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:01.511554 26281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:50:01.516888 26281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.116.212:49045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:01.518131 26281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:01.519197 26281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:01.530088 26288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:01.530231 26290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:01.530560 26287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:01.530870 26281 server_base.cc:1034] running on GCE node
I20260619 06:50:01.531805 26281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:01.532626 26281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:01.533849 26281 hybrid_clock.cc:648] HybridClock initialized: now 1781851801533803 us; error 47 us; skew 500 ppm
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:01.537333 26281 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:50:01.538296 26281 webserver.cc:460] Webserver started at http://127.14.116.194:39379/ using document root <none> and password file <none>
I20260619 06:50:01.538889 26281 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:01.538980 26281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:01.539237 26281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:01.556213 26281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "e69498fd7b0a4c9886f768393ad8e62c"
format_stamp: "Formatted at 2026-06-19 06:50:01 on dist-test-slave-9x3b"
server_key: "b353e96372ce5454c7de651e0bc33e52"
server_key_iv: "1bf05689e66b836cf7672cef0b2a6f7e"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:01.557089 26281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "e69498fd7b0a4c9886f768393ad8e62c"
format_stamp: "Formatted at 2026-06-19 06:50:01 on dist-test-slave-9x3b"
server_key: "b353e96372ce5454c7de651e0bc33e52"
server_key_iv: "1bf05689e66b836cf7672cef0b2a6f7e"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:01.562886 26281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:50:01.585094 26297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:01.586819 26281 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.000s
I20260619 06:50:01.586975 26281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "e69498fd7b0a4c9886f768393ad8e62c"
format_stamp: "Formatted at 2026-06-19 06:50:01 on dist-test-slave-9x3b"
server_key: "b353e96372ce5454c7de651e0bc33e52"
server_key_iv: "1bf05689e66b836cf7672cef0b2a6f7e"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:01.587153 26281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:01.617182 26281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:01.620980 26281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:01.621182 26281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:01.621933 26281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:01.623032 26281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:01.623104 26281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:01.623171 26281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:01.623207 26281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:01.635317 26281 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:36587
I20260619 06:50:01.636597 26281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:50:01.645361 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 26281
I20260619 06:50:01.645505 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:50:01.645817 14803 external_mini_cluster.cc:1428] Setting key 9979c34958e47e7eedf44f3421e91478
I20260619 06:50:01.651204 26410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:36587 every 8 connection(s)
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:50:01.672633 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:01.650886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:32807 (local address 127.14.116.254:39421)
0619 06:50:01.651026 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:01.651030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:01.652871 (+ 1841us) server_negotiation.cc:407] Connection header received
0619 06:50:01.654786 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:01.654790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:01.654849 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:01.654909 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:01.656735 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:01.657570 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:01.659311 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:01.659701 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:01.659786 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:01.663316 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:01.663321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:01.663324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:01.663368 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:01.665113 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:01.670397 (+ 5284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:01.670402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:01.670404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:01.670522 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:01.671046 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:01.671050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:01.671052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:01.671314 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:01.671430 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:01.672275 (+ 845us) server_negotiation.cc:299] Negotiation successful
0619 06:50:01.672449 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:50:01.674714 26414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:01.649675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:39421 (local address 127.14.116.194:32807)
0619 06:50:01.652691 (+ 3016us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:01.652757 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:01.654506 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:01.655374 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:01.655384 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:01.655782 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:01.656534 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:01.656556 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:01.657739 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:01.657744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:01.659160 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:01.659171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:01.659999 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:01.660002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:01.660210 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:01.661388 (+ 1178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:01.661410 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:01.663107 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:01.665259 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:01.665266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:01.665285 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:01.665673 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:01.670676 (+ 5003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:01.670679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:01.670682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:01.670922 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:01.671514 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:01.671522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:01.671955 (+ 433us) client_negotiation.cc:769] Sending connection context
0619 06:50:01.673134 (+ 1179us) client_negotiation.cc:240] Negotiation successful
0619 06:50:01.673544 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2723,"thread_start_us":171,"threads_started":1}
I20260619 06:50:01.676393 26411 heartbeater.cc:347] Connected to a master server at 127.14.116.254:39421
I20260619 06:50:01.676656 26411 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:01.677266 26411 heartbeater.cc:511] Master 127.14.116.254:39421 requested a full tablet report, sending...
I20260619 06:50:01.678499 26057 ts_manager.cc:194] Registered new tserver with Master: e69498fd7b0a4c9886f768393ad8e62c (127.14.116.194:36587)
I20260619 06:50:01.679275 26057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:32807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:50:01.704695 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:49045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39421
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:01.855481 26416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:01.855871 26416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:01.855934 26416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:01.861922 26416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:01.861999 26416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:01.862082 26416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:50:01.869879 26416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.116.212:49045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:01.871232 26416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:01.872537 26416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:01.885660 26424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:01.886123 26421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:01.887037 26416 server_base.cc:1034] running on GCE node
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
W20260619 06:50:01.893064 26422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:01.893658 26416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:01.894492 26416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:01.895664 26416 hybrid_clock.cc:648] HybridClock initialized: now 1781851801895596 us; error 88 us; skew 500 ppm
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:01.899240 26416 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:50:01.900367 26416 webserver.cc:460] Webserver started at http://127.14.116.195:34095/ using document root <none> and password file <none>
I20260619 06:50:01.901070 26416 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:01.901151 26416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:01.901371 26416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:50:01 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:01.938822 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "2a6aad8a860747e684de81288176019c"
format_stamp: "Formatted at 2026-06-19 06:50:01 on dist-test-slave-9x3b"
server_key: "2642de68e1f2e2ea7116a635ead0d677"
server_key_iv: "214e828874d125b381613ac03740bb47"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:01.939545 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "2a6aad8a860747e684de81288176019c"
format_stamp: "Formatted at 2026-06-19 06:50:01 on dist-test-slave-9x3b"
server_key: "2642de68e1f2e2ea7116a635ead0d677"
server_key_iv: "214e828874d125b381613ac03740bb47"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:01.944607 26416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 06:50:01.964025 26431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:01.965699 26416 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 06:50:01.965838 26416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "2a6aad8a860747e684de81288176019c"
format_stamp: "Formatted at 2026-06-19 06:50:01 on dist-test-slave-9x3b"
server_key: "2642de68e1f2e2ea7116a635ead0d677"
server_key_iv: "214e828874d125b381613ac03740bb47"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:01.965994 26416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:01.986498 26416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:01.990334 26416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:01.990540 26416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:01.991218 26416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:01.992722 26416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:01.992806 26416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:01.992868 26416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:01.992902 26416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:02.008129 26416 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:38647
I20260619 06:50:02.008260 26544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:38647 every 8 connection(s)
I20260619 06:50:02.009248 26416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:50:02.009765 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 26416
I20260619 06:50:02.009884 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:50:02.010165 14803 external_mini_cluster.cc:1428] Setting key 0c68f442cbd8c8c05b3c8c1fc0fafc5d
Jun 19 06:50:02 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:02.027204 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.012810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:57903 (local address 127.14.116.254:39421)
0619 06:50:02.012924 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:02.012927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:02.014049 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 06:50:02.015236 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:02.015240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:02.015290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:02.015386 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:02.016775 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.017608 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.019236 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.019589 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.019704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.022610 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:02.022614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:02.022617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:02.022643 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:02.024563 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:02.025417 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:02.025420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:02.025422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:02.025520 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:02.025917 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:02.025921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:02.025922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:02.026145 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:02.026213 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:02.026903 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 06:50:02.027040 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 06:50:02.028846 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.013347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:39421 (local address 127.14.116.195:57903)
0619 06:50:02.013898 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:02.013955 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:02.015010 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:02.015592 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:02.015602 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:02.015977 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:02.016626 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.016646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.017745 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.017749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.019097 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.019107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.019762 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.019766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.019960 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.020653 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:02.020673 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:02.022418 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:02.024680 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:02.024686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:02.024703 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:02.025271 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:02.025615 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:02.025618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:02.025620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:02.025803 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:02.026291 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:02.026298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:02.026630 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 06:50:02.027415 (+ 785us) client_negotiation.cc:240] Negotiation successful
0619 06:50:02.027787 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
I20260619 06:50:02.030004 26545 heartbeater.cc:347] Connected to a master server at 127.14.116.254:39421
I20260619 06:50:02.030332 26545 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:02.030900 26545 heartbeater.cc:511] Master 127.14.116.254:39421 requested a full tablet report, sending...
I20260619 06:50:02.032027 26057 ts_manager.cc:194] Registered new tserver with Master: 2a6aad8a860747e684de81288176019c (127.14.116.195:38647)
I20260619 06:50:02.032749 26057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:57903
I20260619 06:50:02.036783 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:50:02.048547 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.039327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39248 (local address 127.14.116.254:39421)
0619 06:50:02.039431 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:02.039435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:02.039696 (+ 261us) server_negotiation.cc:407] Connection header received
0619 06:50:02.039880 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:02.039883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:02.039925 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:02.039972 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:02.040957 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.041729 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.042917 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.043271 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.043382 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.045084 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:02.045088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:02.045092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:02.045117 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:02.046825 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:02.047279 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:02.047282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:02.047284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:02.047376 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:02.047672 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:02.047677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:02.047678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:02.047887 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:02.048024 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:02.048267 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 06:50:02.048381 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:50:02.051705 26057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:50:02.055979 26057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:50:02.074035 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.066046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:38647 (local address 127.0.0.1:55336)
0619 06:50:02.066450 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:02.066499 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:02.066626 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:02.067600 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:02.067640 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:02.067664 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:02.067802 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.067812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.070135 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.070139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.072299 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.072309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.073487 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.073491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.073608 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.073654 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 06:50:02.073742 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:50:02.073861 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":108,"threads_started":1}
I20260619 06:50:02.075691 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.066762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.14.116.195:38647)
0619 06:50:02.067106 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:02.067117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:02.067140 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:50:02.067213 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:02.067219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:02.067423 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:02.068663 (+ 1240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:02.068799 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.070005 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.072477 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.073326 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.074510 (+ 1184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.074643 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:02.074695 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:50:02.074786 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":122,"threads_started":1}
I20260619 06:50:02.078853 26463 tablet_service.cc:1467] Processing CreateTablet for tablet 60083808eb5a4cab98bf2ac5baa7e59c (DEFAULT_TABLE table=test-table [id=4de71a9999d445a8b3f5ea01e3dc3d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:02.079447 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.068503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:36587 (local address 127.0.0.1:45276)
0619 06:50:02.069819 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:02.069832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:02.069913 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:02.070842 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:02.071114 (+ 272us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:02.071131 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:02.071228 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.071234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.076852 (+ 5618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.076856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.077931 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.077940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.079077 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.079080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.079167 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.079190 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:50:02.079251 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 06:50:02.079312 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"thread_start_us":87,"threads_started":1}
I20260619 06:50:02.080054 26463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60083808eb5a4cab98bf2ac5baa7e59c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:02.081012 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.068887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.14.116.194:36587)
0619 06:50:02.069694 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:02.069704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:02.069999 (+ 295us) server_negotiation.cc:407] Connection header received
0619 06:50:02.070072 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:02.070077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:02.070264 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:02.070402 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:02.071395 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.076733 (+ 5338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.078078 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.078806 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.078915 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.079022 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:02.080783 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0619 06:50:02.080879 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":117,"threads_started":1}
I20260619 06:50:02.083680 26334 tablet_service.cc:1467] Processing CreateTablet for tablet 60083808eb5a4cab98bf2ac5baa7e59c (DEFAULT_TABLE table=test-table [id=4de71a9999d445a8b3f5ea01e3dc3d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:02.084666 26334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60083808eb5a4cab98bf2ac5baa7e59c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:02.094514 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.067496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:45205 (local address 127.0.0.1:35528)
0619 06:50:02.069378 (+ 1882us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:02.069392 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:02.069471 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:02.072653 (+ 3182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:02.072656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:02.072674 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:02.072752 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.072757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.074215 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.074218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.075625 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.075636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.094014 (+ 18378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.094018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.094142 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.094170 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:50:02.094267 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:50:02.094358 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1816,"thread_start_us":77,"threads_started":1}
I20260619 06:50:02.094513 26564 tablet_bootstrap.cc:492] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: Bootstrap starting.
I20260619 06:50:02.094976 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.067864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.14.116.193:45205)
0619 06:50:02.070654 (+ 2790us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:02.070660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:02.070680 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:50:02.070748 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:02.070752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:02.071052 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:02.072578 (+ 1526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:02.072954 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.074088 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.079637 (+ 5549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.093884 (+ 14247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.094027 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.094162 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:02.094705 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 06:50:02.094820 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2692,"thread_start_us":84,"threads_started":1}
I20260619 06:50:02.097962 26207 tablet_service.cc:1467] Processing CreateTablet for tablet 60083808eb5a4cab98bf2ac5baa7e59c (DEFAULT_TABLE table=test-table [id=4de71a9999d445a8b3f5ea01e3dc3d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:02.098904 26564 tablet_bootstrap.cc:654] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:02.099053 26207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60083808eb5a4cab98bf2ac5baa7e59c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:02.100180 26563 tablet_bootstrap.cc:492] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: Bootstrap starting.
I20260619 06:50:02.100486 26564 log.cc:824] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:02.102923 26563 tablet_bootstrap.cc:654] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:02.103788 26563 log.cc:824] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:02.105844 26563 tablet_bootstrap.cc:492] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: No bootstrap required, opened a new log
I20260619 06:50:02.106032 26563 ts_tablet_manager.cc:1397] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:50:02.107976 26567 tablet_bootstrap.cc:492] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115: Bootstrap starting.
I20260619 06:50:02.109908 26563 raft_consensus.cc:348] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } }
I20260619 06:50:02.110124 26567 tablet_bootstrap.cc:654] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:02.110185 26563 raft_consensus.cc:374] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:02.110239 26563 raft_consensus.cc:729] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6aad8a860747e684de81288176019c, State: Initialized, Role: FOLLOWER
I20260619 06:50:02.110716 26563 consensus_queue.cc:260] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } }
I20260619 06:50:02.110908 26567 log.cc:824] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:02.111794 26563 ts_tablet_manager.cc:1428] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 06:50:02.112159 26545 heartbeater.cc:503] Master 127.14.116.254:39421 was elected leader, sending a full tablet report...
I20260619 06:50:02.112769 26567 tablet_bootstrap.cc:492] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115: No bootstrap required, opened a new log
I20260619 06:50:02.112979 26567 ts_tablet_manager.cc:1397] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:50:02.117580 26564 tablet_bootstrap.cc:492] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: No bootstrap required, opened a new log
I20260619 06:50:02.117785 26564 ts_tablet_manager.cc:1397] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20260619 06:50:02.117667 26567 raft_consensus.cc:348] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } }
I20260619 06:50:02.117923 26567 raft_consensus.cc:374] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:02.117980 26567 raft_consensus.cc:729] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2d67132c6f4b559a5f8a0bd0cce115, State: Initialized, Role: FOLLOWER
I20260619 06:50:02.118402 26567 consensus_queue.cc:260] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } }
I20260619 06:50:02.119009 26273 heartbeater.cc:503] Master 127.14.116.254:39421 was elected leader, sending a full tablet report...
I20260619 06:50:02.119390 26567 ts_tablet_manager.cc:1428] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 06:50:02.121655 26564 raft_consensus.cc:348] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } }
I20260619 06:50:02.121914 26564 raft_consensus.cc:374] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:02.121984 26564 raft_consensus.cc:729] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69498fd7b0a4c9886f768393ad8e62c, State: Initialized, Role: FOLLOWER
I20260619 06:50:02.122391 26564 consensus_queue.cc:260] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } }
I20260619 06:50:02.123019 26411 heartbeater.cc:503] Master 127.14.116.254:39421 was elected leader, sending a full tablet report...
I20260619 06:50:02.123363 26564 ts_tablet_manager.cc:1428] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 06:50:02.152200 26412 tablet.cc:2369] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:50:02 dist-test-slave-9x3b krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851801, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
W20260619 06:50:02.262188 26546 tablet.cc:2369] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:50:02.308765 26274 tablet.cc:2369] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:50:02.350279 26571 raft_consensus.cc:482] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:50:02.350548 26571 raft_consensus.cc:504] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } }
I20260619 06:50:02.352027 26571 leader_election.cc:288] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a6aad8a860747e684de81288176019c (127.14.116.195:38647), 9c2d67132c6f4b559a5f8a0bd0cce115 (127.14.116.193:45205)
I20260619 06:50:02.359820 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.352957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:38221 (local address 127.14.116.193:45205)
0619 06:50:02.353060 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:02.353064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:02.354180 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 06:50:02.354422 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:02.354426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:02.354471 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:02.354516 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:02.354907 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.355726 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.357360 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.357984 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.358079 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.358131 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:02.359572 (+ 1441us) server_negotiation.cc:299] Negotiation successful
0619 06:50:02.359688 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 06:50:02.360256 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.352859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:45205 (local address 127.14.116.194:38221)
0619 06:50:02.354082 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:02.354095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:02.354298 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:02.354643 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:02.354645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:02.354660 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:02.354763 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.354769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.355874 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.355877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.357206 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.357217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.359312 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.359316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.359421 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.359447 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:50:02.360037 (+ 590us) client_negotiation.cc:240] Negotiation successful
0619 06:50:02.360113 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"mutex_wait_us":18,"thread_start_us":93,"threads_started":1}
I20260619 06:50:02.360301 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.352595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:43983 (local address 127.14.116.195:38647)
0619 06:50:02.352731 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:02.352734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:02.353775 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 06:50:02.354026 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:02.354030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:02.354086 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:02.354180 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:02.354605 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.355342 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.358968 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.359564 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.359692 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.359743 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:02.360094 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 06:50:02.360174 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 06:50:02.360710 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.352442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:38647 (local address 127.14.116.194:43983)
0619 06:50:02.353221 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:02.353237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:02.353877 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:02.354320 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:02.354323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:02.354343 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:02.354465 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.354472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.355520 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.355523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.357783 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:02.357793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.359837 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.359841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:02.359943 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.359967 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:50:02.360500 (+ 533us) client_negotiation.cc:240] Negotiation successful
0619 06:50:02.360575 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260619 06:50:02.360882 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60083808eb5a4cab98bf2ac5baa7e59c" candidate_uuid: "e69498fd7b0a4c9886f768393ad8e62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" is_pre_election: true
I20260619 06:50:02.361217 26227 raft_consensus.cc:2454] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69498fd7b0a4c9886f768393ad8e62c in term 0.
I20260619 06:50:02.361238 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60083808eb5a4cab98bf2ac5baa7e59c" candidate_uuid: "e69498fd7b0a4c9886f768393ad8e62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6aad8a860747e684de81288176019c" is_pre_election: true
I20260619 06:50:02.361498 26498 raft_consensus.cc:2454] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69498fd7b0a4c9886f768393ad8e62c in term 0.
I20260619 06:50:02.361716 26298 leader_election.cc:302] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c2d67132c6f4b559a5f8a0bd0cce115, e69498fd7b0a4c9886f768393ad8e62c; no voters:
I20260619 06:50:02.362145 26571 raft_consensus.cc:2781] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:50:02.362242 26571 raft_consensus.cc:482] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:50:02.362296 26571 raft_consensus.cc:3037] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:02.363296 26571 raft_consensus.cc:504] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } }
I20260619 06:50:02.363691 26571 leader_election.cc:288] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [CANDIDATE]: Term 1 election: Requested vote from peers 2a6aad8a860747e684de81288176019c (127.14.116.195:38647), 9c2d67132c6f4b559a5f8a0bd0cce115 (127.14.116.193:45205)
I20260619 06:50:02.363996 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60083808eb5a4cab98bf2ac5baa7e59c" candidate_uuid: "e69498fd7b0a4c9886f768393ad8e62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6aad8a860747e684de81288176019c"
I20260619 06:50:02.364049 26227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60083808eb5a4cab98bf2ac5baa7e59c" candidate_uuid: "e69498fd7b0a4c9886f768393ad8e62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115"
I20260619 06:50:02.364107 26498 raft_consensus.cc:3037] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:02.364142 26227 raft_consensus.cc:3037] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:02.365265 26498 raft_consensus.cc:2454] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69498fd7b0a4c9886f768393ad8e62c in term 1.
I20260619 06:50:02.365267 26227 raft_consensus.cc:2454] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69498fd7b0a4c9886f768393ad8e62c in term 1.
I20260619 06:50:02.365574 26298 leader_election.cc:302] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c2d67132c6f4b559a5f8a0bd0cce115, e69498fd7b0a4c9886f768393ad8e62c; no voters:
I20260619 06:50:02.365772 26571 raft_consensus.cc:2781] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:50:02.366029 26571 raft_consensus.cc:686] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 1 LEADER]: Becoming Leader. State: Replica: e69498fd7b0a4c9886f768393ad8e62c, State: Running, Role: LEADER
I20260619 06:50:02.366358 26571 consensus_queue.cc:237] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } }
I20260619 06:50:02.369725 26061 catalog_manager.cc:5495] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c reported cstate change: term changed from 0 to 1, leader changed from <none> to e69498fd7b0a4c9886f768393ad8e62c (127.14.116.194). New cstate: current_term: 1 leader_uuid: "e69498fd7b0a4c9886f768393ad8e62c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e69498fd7b0a4c9886f768393ad8e62c" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36587 } health_report { overall_health: HEALTHY } } }
I20260619 06:50:02.438084 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.434401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.14.116.194:36587)
0619 06:50:02.434530 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:02.434534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:02.434550 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:50:02.434743 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:02.434745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:02.434800 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:02.434920 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:50:02.435174 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.435739 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.436592 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:02.436813 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:02.436932 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:02.437251 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:50:02.437699 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:50:02.437833 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:02.437885 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:50:02.437940 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 06:50:02.443656 26227 raft_consensus.cc:1261] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115 [term 1 FOLLOWER]: Refusing update from remote peer e69498fd7b0a4c9886f768393ad8e62c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:02.443657 26498 raft_consensus.cc:1261] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [term 1 FOLLOWER]: Refusing update from remote peer e69498fd7b0a4c9886f768393ad8e62c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:02.444350 26576 consensus_queue.cc:1035] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a6aad8a860747e684de81288176019c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:02.444550 26571 consensus_queue.cc:1035] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c2d67132c6f4b559a5f8a0bd0cce115" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:02.451689 26586 mvcc.cc:205] Tried to move back new op lower bound from 7298464982802055168 to 7298464982495383552. Current Snapshot: MvccSnapshot[applied={T|T < 7298464982802055168}]
I20260619 06:50:02.451997 26587 mvcc.cc:205] Tried to move back new op lower bound from 7298464982802055168 to 7298464982495383552. Current Snapshot: MvccSnapshot[applied={T|T < 7298464982802055168}]
I20260619 06:50:02.461119 26057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:50:02.461256 26057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:50:02.463507 26057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b442dd9618d412cbe2672e8a24c1ca6: Sending DeleteTablet for 3 replicas of tablet 60083808eb5a4cab98bf2ac5baa7e59c
I20260619 06:50:02.464362 26463 tablet_service.cc:1514] Processing DeleteTablet for tablet 60083808eb5a4cab98bf2ac5baa7e59c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:02 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:55336
I20260619 06:50:02.464408 26334 tablet_service.cc:1514] Processing DeleteTablet for tablet 60083808eb5a4cab98bf2ac5baa7e59c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:02 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:45276
I20260619 06:50:02.464408 26207 tablet_service.cc:1514] Processing DeleteTablet for tablet 60083808eb5a4cab98bf2ac5baa7e59c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:02 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:35528
I20260619 06:50:02.464855 26599 tablet_replica.cc:330] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115: stopping tablet replica
I20260619 06:50:02.465152 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 26142
I20260619 06:50:02.465150 26599 raft_consensus.cc:2229] T 60083808eb5a4cab98bf2ac5baa7e59c P 9c2d67132c6f4b559a5f8a0bd0cce115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:50:02.465745 26598 tablet_replica.cc:330] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: stopping tablet replica
I20260619 06:50:02.466028 26598 raft_consensus.cc:2229] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:50:02.466429 26598 raft_consensus.cc:2258] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:02.466962 26597 tablet_replica.cc:330] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: stopping tablet replica
I20260619 06:50:02.467217 26597 raft_consensus.cc:2229] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:50:02.467489 26597 raft_consensus.cc:2258] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:02.467988 26598 ts_tablet_manager.cc:1905] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:02.468791 26597 ts_tablet_manager.cc:1905] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:02.470649 26598 ts_tablet_manager.cc:1918] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:02.470741 26598 log.cc:1196] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/60083808eb5a4cab98bf2ac5baa7e59c
I20260619 06:50:02.471037 26598 ts_tablet_manager.cc:1939] T 60083808eb5a4cab98bf2ac5baa7e59c P e69498fd7b0a4c9886f768393ad8e62c: Deleting consensus metadata
I20260619 06:50:02.471570 26597 ts_tablet_manager.cc:1918] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:02.471683 26597 log.cc:1196] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/60083808eb5a4cab98bf2ac5baa7e59c
I20260619 06:50:02.472004 26597 ts_tablet_manager.cc:1939] T 60083808eb5a4cab98bf2ac5baa7e59c P 2a6aad8a860747e684de81288176019c: Deleting consensus metadata
W20260619 06:50:02.472178 26045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:45205 (error 108)
I20260619 06:50:02.472258 26046 catalog_manager.cc:4841] TS e69498fd7b0a4c9886f768393ad8e62c (127.14.116.194:36587): tablet 60083808eb5a4cab98bf2ac5baa7e59c (table test-table [id=4de71a9999d445a8b3f5ea01e3dc3d33]) successfully deleted
I20260619 06:50:02.472635 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 26281
I20260619 06:50:02.473742 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:02.473446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:45205 (local address 127.0.0.1:35538)
0619 06:50:02.473588 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:02.473659 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 06:50:02.473946 26045 catalog_manager.cc:4584] TS 9c2d67132c6f4b559a5f8a0bd0cce115 (127.14.116.193:45205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60083808eb5a4cab98bf2ac5baa7e59c: Network error: Client connection negotiation failed: client connection to 127.14.116.193:45205: connect: Connection refused (error 111)
I20260619 06:50:02.474009 26046 catalog_manager.cc:4841] TS 2a6aad8a860747e684de81288176019c (127.14.116.195:38647): tablet 60083808eb5a4cab98bf2ac5baa7e59c (table test-table [id=4de71a9999d445a8b3f5ea01e3dc3d33]) successfully deleted
I20260619 06:50:02.478314 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 26416
I20260619 06:50:02.483355 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 26029
I20260619 06:50:02.489506 14803 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:50:02.824323 14803 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:50:02.824469 14803 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:50:03.168699 14803 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:50:03.168 UTC [25615] LOG: received smart shutdown request
2026-06-19 06:50:03.172 UTC [25615] LOG: background worker "logical replication launcher" (PID 25624) exited with exit code 1
2026-06-19 06:50:03.172 UTC [25619] LOG: shutting down
2026-06-19 06:50:03.307 UTC [25615] LOG: database system is shut down
2026-06-19T06:50:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45052 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:50:03 dist-test-slave-9x3b krb5kdc[26609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:50:03 dist-test-slave-9x3b krb5kdc[26609](info): set up 2 sockets
Jun 19 06:50:03 dist-test-slave-9x3b krb5kdc[26609](info): commencing operation
krb5kdc: starting...
W20260619 06:50:04.463744 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:50:04 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:50:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/rangeradmin_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/rangeradmin_127.14.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/rangerlookup_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/rangerlookup_127.14.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/HTTP_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/HTTP_127.14.116.212@KRBTEST.COM.keytab.
I20260619 06:50:04.626350 14803 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2eB_D1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/postgres -l logfile start
2026-06-19 06:50:06.936 UTC [26639] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:50:06.936 UTC [26639] LOG: listening on IPv4 address "127.14.116.212", port 43759
2026-06-19 06:50:06.939 UTC [26639] LOG: listening on Unix socket "/tmp/.s.PGSQL.43759"
2026-06-19 06:50:06.960 UTC [26642] LOG: database system was shut down at 2026-06-19 06:50:05 UTC
2026-06-19 06:50:06.963 UTC [26639] LOG: database system is ready to accept connections
I20260619 06:50:07.919386 14803 mini_postgres.cc:96] Postgres bound to 43759
2026-06-19 06:50:07.925 UTC [26650] FATAL: database "slave" does not exist
127.14.116.212:43759 - accepting connections
I20260619 06:50:07.925889 14803 mini_ranger.cc:161] Starting Ranger...
I20260619 06:50:07.940261 14803 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:50:08.367149 14803 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:50:08.367236 14803 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-admin
2026-06-19 06:50:08,642 [I] DB FLAVOR :POSTGRES
2026-06-19 06:50:08,643 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:50:08,643 [I] Checking connection..
2026-06-19 06:50:08,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:50:08,881 [I] Checking connection passed.
2026-06-19 06:50:08,882 [I] --------- Verifying version history table ---------
2026-06-19 06:50:08,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:50:09,109 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:50:09,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:50:09,345 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:50:09,345 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:50:09,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:50:09.594 UTC [26714] WARNING: there is no transaction in progress
2026-06-19 06:50:09,600 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:50:09,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:50:09,827 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:50:09,827 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:50:09,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:50:10,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:50:10,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9x3b.c.gcp-upstream.internal','N') ;"
2026-06-19 06:50:10,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:50:10,821 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:50:10,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:50:11,045 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:50:11,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:50:11,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:50:11,518 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:50:11,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:50:12.426 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.442 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.457 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.472 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.490 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.509 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.571 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.577 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.590 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.600 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.609 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.622 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.631 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.636 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.642 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:12.648 UTC [26822] WARNING: there is no transaction in progress
2026-06-19 06:50:13,201 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:50:13,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:50:13,448 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:50:13,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:50:13,692 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:50:13,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:50:13,947 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:50:13,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:50:14,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:50:14,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9x3b.c.gcp-upstream.internal';"
2026-06-19 06:50:14,676 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:50:14,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:50:14,913 [I] DB_PATCHES have already been applied
I20260619 06:50:14.919047 14803 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-admin:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2eB_D1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:50:14.919106 14803 mini_ranger.cc:193] Using host: 127.14.116.212
I20260619 06:50:14.922353 14803 mini_ranger.cc:227] Ranger admin URL: http://127.14.116.212:41177
Jun 19, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:50:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:50:16.017458 14803 mini_ranger.cc:160] Time spent starting Ranger: real 8.092s user 0.000s sys 0.009s
Jun 19, 2026 6:50:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:50:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 6:50:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:50:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:50:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:50:36.517488 14803 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.116.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.116.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/rangerkms_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.116.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/rangerkms_127.14.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.116.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.116.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.116.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/HTTP_127.14.116.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.116.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/HTTP_127.14.116.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:50:36 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851836, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:50:36.610167 14803 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:50:36.619961 14803 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:50:37.375270 14803 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:50:37.375370 14803 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-kms
I20260619 06:50:37.375393 14803 mini_ranger_kms.cc:227] Using postgres at 127.14.116.212:43759
2026-06-19 06:50:37,539 [I] DB FLAVOR :POSTGRES
2026-06-19 06:50:37,539 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:50:37,539 [I] Checking connection
2026-06-19 06:50:37,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:50:37,765 [I] connection success
2026-06-19 06:50:37,765 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:50:37,765 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:50:37,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:50:37,992 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:50:37,992 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:50:37,992 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:50:37,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.116.212:43759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:50:38,232 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:50:38.657162 14803 mini_ranger_kms.cc:310] Created kms service
I20260619 06:50:38.828116 14803 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:50:38.868753 14803 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:50:38 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851836, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:39.241466 14803 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:50:39.241616 14803 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-kms:/tmp/dist-test-task2eB_D1/build/debug/bin/postgresql.jar:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2eB_D1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:50:39.241650 14803 mini_ranger_kms.cc:240] Using host: 127.14.116.212
I20260619 06:50:39.245329 14803 mini_ranger_kms.cc:276] Ranger KMS PID: 27002
I20260619 06:50:39.245409 14803 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.116.212:35947
Jun 19, 2026 6:50:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:50:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:50:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:50:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:50:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:50:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 06:50:41.441784 14803 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.832s user 0.015s sys 0.133s
I20260619 06:50:41.441910 14803 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:50:41.441977 14803 mini_ranger_kms.cc:401] 127.14.116.212:35947/kms/v1/keys
Jun 19 06:50:41 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851841, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 06:50:41 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851841, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.212@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
Jun 19 06:50:42 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851841, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.116.212@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
Jun 19 06:50:44 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:50:44.459054 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:34189
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:34189
--ranger_config_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:35947/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:44.566074 27053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:44.566306 27053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:44.566356 27053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:44.570334 27053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:44.570382 27053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:44.570402 27053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:44.570459 27053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:44.570482 27053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:44.575891 27053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.116.212:35947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:34189
--ranger_config_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:44.576823 27053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:44.577788 27053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:44.586297 27059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:44.586303 27061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:44.586378 27053 server_base.cc:1034] running on GCE node
W20260619 06:50:44.586303 27058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:44.587090 27053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:44.588124 27053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:44.589316 27053 hybrid_clock.cc:648] HybridClock initialized: now 1781851844589288 us; error 38 us; skew 500 ppm
Jun 19 06:50:44 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851844, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:44.592036 27053 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:50:44.592864 27053 webserver.cc:460] Webserver started at http://127.14.116.254:35419/ using document root <none> and password file <none>
I20260619 06:50:44.593391 27053 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:44.593472 27053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:44.593677 27053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:50:44 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851844, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:44.647061 27053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "b04935251cfc4801b1c68d4e2cc24c82"
format_stamp: "Formatted at 2026-06-19 06:50:44 on dist-test-slave-9x3b"
server_key: "4ff64c91776345dfbeef5b6f6b0297dc"
server_key_iv: "dca8e0e256e556d566e82622554d78f4"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:44.647756 27053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "b04935251cfc4801b1c68d4e2cc24c82"
format_stamp: "Formatted at 2026-06-19 06:50:44 on dist-test-slave-9x3b"
server_key: "4ff64c91776345dfbeef5b6f6b0297dc"
server_key_iv: "dca8e0e256e556d566e82622554d78f4"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:44.653041 27053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:50:44.668470 27068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:44.669880 27053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:50:44.670027 27053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "b04935251cfc4801b1c68d4e2cc24c82"
format_stamp: "Formatted at 2026-06-19 06:50:44 on dist-test-slave-9x3b"
server_key: "4ff64c91776345dfbeef5b6f6b0297dc"
server_key_iv: "dca8e0e256e556d566e82622554d78f4"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:44.670171 27053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:44.681680 27053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:44.684455 27053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:44.684876 27053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:44.692536 27053 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:34189
I20260619 06:50:44.692591 27120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:34189 every 8 connection(s)
I20260619 06:50:44.693478 27053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:50:44.695601 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 27053
I20260619 06:50:44.695765 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:50:44.696053 14803 external_mini_cluster.cc:1428] Setting key 65dc66bb5d496ff594c571454128bdf6
I20260619 06:50:44.696758 27121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:44.703487 27121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Bootstrap starting.
Jun 19 06:50:44 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:44.705927 27121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:44.706866 27121 log.cc:824] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:44.709213 27121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: No bootstrap required, opened a new log
I20260619 06:50:44.711004 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:44.697346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.14.116.254:34189)
0619 06:50:44.697723 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:44.697742 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:44.697985 (+ 243us) server_negotiation.cc:407] Connection header received
0619 06:50:44.698784 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:44.698806 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:44.699159 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:44.699484 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:44.700310 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:44.701254 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:44.702117 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:44.702573 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:44.702751 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:44.705477 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:44.705482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:44.705494 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:44.705519 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:44.707956 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:44.708518 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:44.708524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:44.708529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:44.708661 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:44.708938 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:44.708943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:44.708947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:44.709439 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:44.709687 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:44.709828 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 06:50:44.710090 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
I20260619 06:50:44.712639 27121 raft_consensus.cc:348] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } }
I20260619 06:50:44.712847 27121 raft_consensus.cc:374] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:44.712949 27121 raft_consensus.cc:729] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04935251cfc4801b1c68d4e2cc24c82, State: Initialized, Role: FOLLOWER
I20260619 06:50:44.713438 27121 consensus_queue.cc:260] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } }
I20260619 06:50:44.713595 27121 raft_consensus.cc:388] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:50:44.713660 27121 raft_consensus.cc:482] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:50:44.713757 27121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:44.714608 27121 raft_consensus.cc:504] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } }
I20260619 06:50:44.714938 27121 leader_election.cc:302] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b04935251cfc4801b1c68d4e2cc24c82; no voters:
I20260619 06:50:44.715243 27121 leader_election.cc:288] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:50:44.715348 27126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:50:44.715567 27126 raft_consensus.cc:686] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 1 LEADER]: Becoming Leader. State: Replica: b04935251cfc4801b1c68d4e2cc24c82, State: Running, Role: LEADER
I20260619 06:50:44.715955 27126 consensus_queue.cc:237] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } }
I20260619 06:50:44.716478 27121 sys_catalog.cc:564] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:50:44.717567 27128 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04935251cfc4801b1c68d4e2cc24c82. Latest consensus state: current_term: 1 leader_uuid: "b04935251cfc4801b1c68d4e2cc24c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } } }
I20260619 06:50:44.717682 27128 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:44.718070 27135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:50:44.718039 27127 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b04935251cfc4801b1c68d4e2cc24c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } } }
I20260619 06:50:44.718647 27127 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:44.720770 27135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:50:44.721160 27121 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:50:44.729347 27135 catalog_manager.cc:1347] Generated new cluster ID: e817a5703f8041f8b1e7511c626bbb4c
I20260619 06:50:44.729415 27135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:50:44.739362 27135 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:50:44.740540 27135 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:50:44.749462 27135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Generated new TSK 0
I20260619 06:50:44.750061 27135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:50:45.428702 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34189
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:45.543959 27166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:45.544263 27166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:45.544337 27166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:45.550208 27166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:45.550323 27166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:45.550436 27166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:50:45.556960 27166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.116.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:45.558019 27166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:45.559010 27166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:45.568014 27174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:45.568377 27171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:45.568523 27172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:45.568857 27166 server_base.cc:1034] running on GCE node
I20260619 06:50:45.569231 27166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:45.569833 27166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:45.571015 27166 hybrid_clock.cc:648] HybridClock initialized: now 1781851845570989 us; error 45 us; skew 500 ppm
Jun 19 06:50:45 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:45.577154 27166 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:50:45.578284 27166 webserver.cc:460] Webserver started at http://127.14.116.193:37841/ using document root <none> and password file <none>
I20260619 06:50:45.578969 27166 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:45.579085 27166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:45.579337 27166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:50:45 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:45.594717 27166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "c88d62afe7cb42e5b6d71ca6e39f668f"
format_stamp: "Formatted at 2026-06-19 06:50:45 on dist-test-slave-9x3b"
server_key: "fad32184759fe9f44bd9bae244dd692b"
server_key_iv: "79de8c977658fa38e1632946a079bbea"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:45.595506 27166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "c88d62afe7cb42e5b6d71ca6e39f668f"
format_stamp: "Formatted at 2026-06-19 06:50:45 on dist-test-slave-9x3b"
server_key: "fad32184759fe9f44bd9bae244dd692b"
server_key_iv: "79de8c977658fa38e1632946a079bbea"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:45.602910 27166 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260619 06:50:45.622428 27183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:45.624135 27166 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260619 06:50:45.624264 27166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "c88d62afe7cb42e5b6d71ca6e39f668f"
format_stamp: "Formatted at 2026-06-19 06:50:45 on dist-test-slave-9x3b"
server_key: "fad32184759fe9f44bd9bae244dd692b"
server_key_iv: "79de8c977658fa38e1632946a079bbea"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:45.624406 27166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:45.649086 27166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:45.657477 27166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:45.657711 27166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:45.658521 27166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:45.659543 27166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:45.659679 27166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:45.659754 27166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:45.659778 27166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:45.670845 27166 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:38115
I20260619 06:50:45.673499 27296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:38115 every 8 connection(s)
I20260619 06:50:45.673933 27166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:50:45.678642 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 27166
I20260619 06:50:45.678769 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:50:45.679090 14803 external_mini_cluster.cc:1428] Setting key d0f90bae5fb5c3de61f390c86ef74301
Jun 19 06:50:45 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
I20260619 06:50:45.693092 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:45.677244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:44413 (local address 127.14.116.254:34189)
0619 06:50:45.677605 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:45.677614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:45.678228 (+ 614us) server_negotiation.cc:407] Connection header received
0619 06:50:45.680663 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:45.680667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:45.680756 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:45.680824 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:45.682534 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:45.683388 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:45.685208 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:45.685584 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:45.685692 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:45.688656 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:45.688659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:45.688662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:45.688688 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:45.690595 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:45.691220 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:45.691223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:45.691225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:45.691315 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:45.691722 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:45.691726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:45.691727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:45.691952 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:45.692036 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:45.692811 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 06:50:45.692936 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":108,"threads_started":1}
I20260619 06:50:45.694003 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:45.676574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:34189 (local address 127.14.116.193:44413)
0619 06:50:45.677972 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:45.678027 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:45.679025 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:45.681225 (+ 2200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:45.681234 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:45.681571 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:45.682277 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:45.682297 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:45.683550 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:45.683553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:45.685052 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:45.685067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:45.685861 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:45.685865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:45.686072 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:45.686721 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:45.686741 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:45.688474 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:45.690738 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:45.690743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:45.690757 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:45.691104 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:45.691420 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:45.691422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:45.691425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:45.691586 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:45.692099 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:45.692105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:45.692496 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 06:50:45.692731 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 06:50:45.693048 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":127,"threads_started":1}
I20260619 06:50:45.695448 27297 heartbeater.cc:347] Connected to a master server at 127.14.116.254:34189
I20260619 06:50:45.695691 27297 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:45.696225 27297 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
I20260619 06:50:45.697798 27085 ts_manager.cc:194] Registered new tserver with Master: c88d62afe7cb42e5b6d71ca6e39f668f (127.14.116.193:38115)
I20260619 06:50:45.699297 27085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:44413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:50:45.724504 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34189
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:50:45 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:50:45.878530 27305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:45.878860 27305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:45.878922 27305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:45.884949 27305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:45.885023 27305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:45.885108 27305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:50:45.892604 27305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.116.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:45.893805 27305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:45.894876 27305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:45.904798 27311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:45.904810 27312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:45.905138 27314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:45.905282 27305 server_base.cc:1034] running on GCE node
I20260619 06:50:45.905705 27305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:45.906493 27305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:45.907687 27305 hybrid_clock.cc:648] HybridClock initialized: now 1781851845907621 us; error 83 us; skew 500 ppm
Jun 19 06:50:45 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:45.911067 27305 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:50:45.912077 27305 webserver.cc:460] Webserver started at http://127.14.116.194:44147/ using document root <none> and password file <none>
I20260619 06:50:45.912719 27305 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:45.912801 27305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:45.913007 27305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:50:45 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:45.936945 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "9240cb0ab05a4631a6b2c517be71a0f7"
format_stamp: "Formatted at 2026-06-19 06:50:45 on dist-test-slave-9x3b"
server_key: "c1a526851530163d076c1b419f4c7537"
server_key_iv: "4ec02c532d7a00fa5f81362305850a91"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:45.937671 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "9240cb0ab05a4631a6b2c517be71a0f7"
format_stamp: "Formatted at 2026-06-19 06:50:45 on dist-test-slave-9x3b"
server_key: "c1a526851530163d076c1b419f4c7537"
server_key_iv: "4ec02c532d7a00fa5f81362305850a91"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:45.942185 27305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:50:45.959466 27321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:45.960898 27305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:50:45.961035 27305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "9240cb0ab05a4631a6b2c517be71a0f7"
format_stamp: "Formatted at 2026-06-19 06:50:45 on dist-test-slave-9x3b"
server_key: "c1a526851530163d076c1b419f4c7537"
server_key_iv: "4ec02c532d7a00fa5f81362305850a91"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:45.961177 27305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:45.986371 27305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:45.990125 27305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:45.990367 27305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:45.991115 27305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:45.992087 27305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:45.992156 27305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:45.992219 27305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:45.992249 27305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:46.003672 27305 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:39077
I20260619 06:50:46.003886 27434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:39077 every 8 connection(s)
I20260619 06:50:46.004787 27305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:50:46.014518 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 27305
I20260619 06:50:46.014665 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:50:46.014981 14803 external_mini_cluster.cc:1428] Setting key eb8f0caf3f1a3c172d46316bb5665f1d
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:50:46.025135 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:46.006600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:42131 (local address 127.14.116.254:34189)
0619 06:50:46.006742 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:46.006748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:46.011999 (+ 5251us) server_negotiation.cc:407] Connection header received
0619 06:50:46.013117 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:46.013119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:46.013166 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:46.013249 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:46.014735 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.015488 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.017177 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.017627 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.017750 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:46.020734 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:46.020738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:46.020741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:46.020769 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:46.022599 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:46.023259 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:46.023263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:46.023265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:46.023356 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:46.023796 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:46.023800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:46.023802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:46.024038 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:46.024140 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:46.024815 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 06:50:46.024965 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:50:46.026016 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:46.006962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:34189 (local address 127.14.116.194:42131)
0619 06:50:46.011804 (+ 4842us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:46.011866 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:46.012874 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:46.013458 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:46.013468 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:46.013816 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:46.014514 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:46.014535 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.015671 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.015676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:46.017050 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:46.017062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.017790 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.017795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:46.017998 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:46.018690 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:46.018712 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:46.020548 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:46.022728 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:46.022735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:46.022752 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:46.023125 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:46.023458 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:46.023463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:46.023465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:46.023678 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:46.024163 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:46.024172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:46.024526 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 06:50:46.024730 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 06:50:46.025095 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":143,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:50:46.027275 27435 heartbeater.cc:347] Connected to a master server at 127.14.116.254:34189
I20260619 06:50:46.027514 27435 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:46.028133 27435 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
I20260619 06:50:46.029384 27085 ts_manager.cc:194] Registered new tserver with Master: 9240cb0ab05a4631a6b2c517be71a0f7 (127.14.116.194:39077)
I20260619 06:50:46.030184 27085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:42131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:50:46.048051 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34189
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
W20260619 06:50:46.210391 27440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:46.210743 27440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:46.210803 27440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:46.216810 27440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:46.216883 27440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:46.216966 27440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:50:46.224575 27440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.116.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:46.225833 27440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:46.227072 27440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:46.238471 27445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:46.238811 27448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:46.238863 27446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:46.239360 27440 server_base.cc:1034] running on GCE node
I20260619 06:50:46.239852 27440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:46.240583 27440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:46.241817 27440 hybrid_clock.cc:648] HybridClock initialized: now 1781851846241792 us; error 41 us; skew 500 ppm
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851846, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:46.245290 27440 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:50:46.246348 27440 webserver.cc:460] Webserver started at http://127.14.116.195:41869/ using document root <none> and password file <none>
I20260619 06:50:46.247051 27440 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:46.247171 27440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:46.247453 27440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851846, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:46.262027 27440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9"
format_stamp: "Formatted at 2026-06-19 06:50:46 on dist-test-slave-9x3b"
server_key: "921cfafb5f3c0580b46208e473550b04"
server_key_iv: "42395bda7120c70160dd16d348e280e1"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:46.262720 27440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9"
format_stamp: "Formatted at 2026-06-19 06:50:46 on dist-test-slave-9x3b"
server_key: "921cfafb5f3c0580b46208e473550b04"
server_key_iv: "42395bda7120c70160dd16d348e280e1"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:46.267470 27440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:50:46.282472 27455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:46.284269 27440 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 06:50:46.284406 27440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9"
format_stamp: "Formatted at 2026-06-19 06:50:46 on dist-test-slave-9x3b"
server_key: "921cfafb5f3c0580b46208e473550b04"
server_key_iv: "42395bda7120c70160dd16d348e280e1"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:46.284531 27440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:46.306842 27440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:46.310355 27440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:46.310529 27440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:46.311148 27440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:46.312103 27440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:46.312158 27440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:46.312206 27440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:46.312233 27440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:46.322530 27440 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:40249
I20260619 06:50:46.322674 27568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:40249 every 8 connection(s)
I20260619 06:50:46.323560 27440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:50:46.328274 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 27440
I20260619 06:50:46.328402 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:50:46.328680 14803 external_mini_cluster.cc:1428] Setting key b836d0d175162faa9e4822ce597f212e
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851846, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:46.341657 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:46.327840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:36261 (local address 127.14.116.254:34189)
0619 06:50:46.327967 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:46.327971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:46.328394 (+ 423us) server_negotiation.cc:407] Connection header received
0619 06:50:46.329766 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:46.329770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:46.329825 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:46.329877 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:46.331578 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.332431 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.334139 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.334509 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.334608 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:46.337448 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:46.337452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:46.337455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:46.337482 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:46.339207 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:46.339828 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:46.339832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:46.339834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:46.339934 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:46.340336 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:46.340340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:46.340342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:46.340578 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:46.340689 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:46.341387 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 06:50:46.341502 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:50:46.343118 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:46.326961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:34189 (local address 127.14.116.195:36261)
0619 06:50:46.328232 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:46.328288 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:46.329460 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:46.330307 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:46.330317 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:46.330755 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:46.331422 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:46.331440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.332581 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.332584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:46.334016 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:46.334027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.334674 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.334679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:46.334868 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:46.335566 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:46.335587 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:46.337260 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:46.339346 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:46.339352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:46.339366 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:46.339740 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:46.340038 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:46.340041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:46.340043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:46.340212 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:46.340693 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:46.340699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:46.341002 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 06:50:46.341846 (+ 844us) client_negotiation.cc:240] Negotiation successful
0619 06:50:46.342148 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"thread_start_us":131,"threads_started":1}
I20260619 06:50:46.344215 27569 heartbeater.cc:347] Connected to a master server at 127.14.116.254:34189
I20260619 06:50:46.344512 27569 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:46.345069 27569 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
I20260619 06:50:46.346148 27085 ts_manager.cc:194] Registered new tserver with Master: c9861ed24c0d4df5a7a6cf8bc3dd15a9 (127.14.116.195:40249)
I20260619 06:50:46.346843 27085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:36261
I20260619 06:50:46.354200 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:50:46.354955 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 27166
I20260619 06:50:46.362471 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 27305
I20260619 06:50:46.370235 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 27440
I20260619 06:50:46.376796 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 27053
I20260619 06:50:46.382633 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:34189
--webserver_interface=127.14.116.254
--webserver_port=35419
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:34189
--ranger_config_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:35947/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.116.212:35947/kms with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:46.489404 27572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:46.489632 27572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:46.489699 27572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:46.493697 27572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:46.493752 27572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:46.493772 27572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:46.493788 27572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:46.493837 27572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:46.499202 27572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.116.212:35947/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:34189
--ranger_config_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:46.500299 27572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:46.501271 27572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:46.509450 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:46.509545 27578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:46.509555 27577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:46.509770 27572 server_base.cc:1034] running on GCE node
I20260619 06:50:46.510324 27572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:46.511333 27572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:46.512516 27572 hybrid_clock.cc:648] HybridClock initialized: now 1781851846512477 us; error 55 us; skew 500 ppm
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851846, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:46.515100 27572 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:50:46.516016 27572 webserver.cc:460] Webserver started at http://127.14.116.254:35419/ using document root <none> and password file <none>
I20260619 06:50:46.516531 27572 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:46.516623 27572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851846, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:46.597982 27572 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:50:46.600420 27588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:46.601649 27572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:50:46.601802 27572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "b04935251cfc4801b1c68d4e2cc24c82"
format_stamp: "Formatted at 2026-06-19 06:50:44 on dist-test-slave-9x3b"
server_key: "4ff64c91776345dfbeef5b6f6b0297dc"
server_key_iv: "dca8e0e256e556d566e82622554d78f4"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:46.602329 27572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:46.626628 27572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:46.629101 27572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:46.629303 27572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:46.636511 27572 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:34189
I20260619 06:50:46.636519 27640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:34189 every 8 connection(s)
I20260619 06:50:46.637511 27572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:50:46.639689 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 27572
I20260619 06:50:46.640218 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:38115
--local_ip_for_outbound_sockets=127.14.116.193
--tserver_master_addrs=127.14.116.254:34189
--webserver_port=37841
--webserver_interface=127.14.116.193
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.116.212:35947/kms with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:50:46.642096 27641 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:50:46.643651 27641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Bootstrap starting.
I20260619 06:50:46.659684 27641 log.cc:824] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:46.666052 27641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:50:46.666455 27641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Bootstrap complete.
I20260619 06:50:46.670459 27641 raft_consensus.cc:348] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } }
I20260619 06:50:46.670830 27641 raft_consensus.cc:729] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04935251cfc4801b1c68d4e2cc24c82, State: Initialized, Role: FOLLOWER
I20260619 06:50:46.671356 27641 consensus_queue.cc:260] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } }
I20260619 06:50:46.671479 27641 raft_consensus.cc:388] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:50:46.671563 27641 raft_consensus.cc:482] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:50:46.671692 27641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:50:46.673501 27641 raft_consensus.cc:504] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } }
I20260619 06:50:46.673830 27641 leader_election.cc:302] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b04935251cfc4801b1c68d4e2cc24c82; no voters:
I20260619 06:50:46.674152 27641 leader_election.cc:288] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:50:46.674283 27645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:50:46.674559 27645 raft_consensus.cc:686] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [term 2 LEADER]: Becoming Leader. State: Replica: b04935251cfc4801b1c68d4e2cc24c82, State: Running, Role: LEADER
I20260619 06:50:46.674926 27645 consensus_queue.cc:237] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } }
I20260619 06:50:46.675491 27641 sys_catalog.cc:564] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:50:46.676465 27646 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04935251cfc4801b1c68d4e2cc24c82. Latest consensus state: current_term: 2 leader_uuid: "b04935251cfc4801b1c68d4e2cc24c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } } }
I20260619 06:50:46.676504 27647 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b04935251cfc4801b1c68d4e2cc24c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04935251cfc4801b1c68d4e2cc24c82" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34189 } } }
I20260619 06:50:46.676595 27646 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:46.676599 27647 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82 [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:46.676932 27654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:50:46.680078 27654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:50:46.680553 27641 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:50:46.680706 27654 catalog_manager.cc:1259] Loaded cluster ID: e817a5703f8041f8b1e7511c626bbb4c
I20260619 06:50:46.680748 27654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:50:46.682976 27654 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:50:46.683597 27654 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Loaded TSK: 0
I20260619 06:50:46.685097 27654 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:50:46.778262 27643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:46.778594 27643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:46.778656 27643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:46.784652 27643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:46.784729 27643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:46.784813 27643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:50:46.792423 27643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.116.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:38115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=37841
--tserver_master_addrs=127.14.116.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:46.793808 27643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:46.795145 27643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:46.806824 27681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:46.807667 27679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:46.807869 27643 server_base.cc:1034] running on GCE node
W20260619 06:50:46.808171 27678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:46.808702 27643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:46.809418 27643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:46.810621 27643 hybrid_clock.cc:648] HybridClock initialized: now 1781851846810604 us; error 30 us; skew 500 ppm
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851846, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:46.814033 27643 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:50:46.815021 27643 webserver.cc:460] Webserver started at http://127.14.116.193:37841/ using document root <none> and password file <none>
I20260619 06:50:46.815685 27643 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:46.815778 27643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851846, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:46.835903 27643 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 06:50:46.839058 27689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:46.841050 27643 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 06:50:46.841226 27643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "c88d62afe7cb42e5b6d71ca6e39f668f"
format_stamp: "Formatted at 2026-06-19 06:50:45 on dist-test-slave-9x3b"
server_key: "fad32184759fe9f44bd9bae244dd692b"
server_key_iv: "79de8c977658fa38e1632946a079bbea"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:46.842001 27643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:46.858223 27643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:46.861598 27643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:46.861824 27643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:46.862489 27643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:46.863512 27643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:46.863629 27643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:46.863754 27643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:46.863828 27643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:46.873636 27643 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:38115
I20260619 06:50:46.874823 27643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:50:46.882519 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 27643
I20260619 06:50:46.882951 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:39077
--local_ip_for_outbound_sockets=127.14.116.194
--tserver_master_addrs=127.14.116.254:34189
--webserver_port=44147
--webserver_interface=127.14.116.194
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.116.212:35947/kms with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:50:46.884611 27802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:38115 every 8 connection(s)
Jun 19 06:50:46 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851846, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:46.908057 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:46.886830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:34189 (local address 127.14.116.193:38517)
0619 06:50:46.889070 (+ 2240us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:46.889118 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:46.890096 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:46.891883 (+ 1787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:46.891892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:46.892219 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:46.892820 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:46.892837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.894683 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.894687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:46.896132 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:46.896146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.897688 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.897694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:46.897930 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:46.898647 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:46.898673 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:46.900638 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:46.903583 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:46.903592 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:46.903633 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:46.904376 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:46.904833 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:46.904838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:46.904842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:46.905023 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:46.906020 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:46.906026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:46.906537 (+ 511us) client_negotiation.cc:769] Sending connection context
0619 06:50:46.906813 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 06:50:46.907107 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":154,"threads_started":1}
I20260619 06:50:46.909780 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:46.888159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:38517 (local address 127.14.116.254:34189)
0619 06:50:46.888806 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:46.888826 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:46.890412 (+ 1586us) server_negotiation.cc:407] Connection header received
0619 06:50:46.890683 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:46.890695 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:46.891081 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:46.891399 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:46.893019 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.894537 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.896273 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:46.897052 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:46.897229 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:46.900879 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:46.900887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:46.900905 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:46.900942 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:46.903435 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:46.904526 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:46.904536 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:46.904544 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:46.904692 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:46.905235 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:46.905242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:46.905246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:46.905825 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:46.905966 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:46.908667 (+ 2701us) server_negotiation.cc:299] Negotiation successful
0619 06:50:46.908937 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":99,"threads_started":1}
I20260619 06:50:46.912892 27803 heartbeater.cc:347] Connected to a master server at 127.14.116.254:34189
I20260619 06:50:46.913108 27803 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:46.913661 27803 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
W20260619 06:50:46.915757 27803 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:34189 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:50:46.915876 27803 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:46.916035 27803 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
I20260619 06:50:46.916870 27803 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:46.917017 27803 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
W20260619 06:50:46.917716 27803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:50:47.025219 27805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:47.025527 27805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:47.025630 27805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:47.029963 27805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:47.030067 27805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:47.030175 27805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:50:47.035473 27805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.116.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:39077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=44147
--tserver_master_addrs=127.14.116.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:47.036535 27805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:47.037508 27805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:47.046687 27813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:47.046967 27815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:47.046705 27812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:47.047314 27805 server_base.cc:1034] running on GCE node
I20260619 06:50:47.047756 27805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:47.048404 27805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:47.049633 27805 hybrid_clock.cc:648] HybridClock initialized: now 1781851847049596 us; error 45 us; skew 500 ppm
Jun 19 06:50:47 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:47.052356 27805 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:50:47.053239 27805 webserver.cc:460] Webserver started at http://127.14.116.194:44147/ using document root <none> and password file <none>
I20260619 06:50:47.053812 27805 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:47.053882 27805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:50:47 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:47.091370 27805 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:50:47.094210 27823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:47.095546 27805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:50:47.095706 27805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "9240cb0ab05a4631a6b2c517be71a0f7"
format_stamp: "Formatted at 2026-06-19 06:50:45 on dist-test-slave-9x3b"
server_key: "c1a526851530163d076c1b419f4c7537"
server_key_iv: "4ec02c532d7a00fa5f81362305850a91"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:47.096251 27805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:47.118340 27805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:47.121699 27805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:47.121913 27805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:47.122541 27805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:47.123415 27805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:47.123484 27805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:47.123580 27805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:47.123646 27805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:47.135013 27805 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:39077
I20260619 06:50:47.135069 27936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:39077 every 8 connection(s)
I20260619 06:50:47.136289 27805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:50:47.140023 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 27805
I20260619 06:50:47.140513 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.116.212:35947/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:40249
--local_ip_for_outbound_sockets=127.14.116.195
--tserver_master_addrs=127.14.116.254:34189
--webserver_port=41869
--webserver_interface=127.14.116.195
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.116.212:35947/kms with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:50:47 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:47.151064 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:47.138186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:35239 (local address 127.14.116.254:34189)
0619 06:50:47.138313 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:47.138317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:47.139265 (+ 948us) server_negotiation.cc:407] Connection header received
0619 06:50:47.140546 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:47.140551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:47.140616 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:47.140681 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:47.142369 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.143183 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.144496 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.144726 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.144835 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:47.147311 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:47.147317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:47.147324 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:47.147347 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:47.148982 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:47.149538 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:47.149541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:47.149543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:47.149624 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:47.149971 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:47.149976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:47.149978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:47.150159 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:47.150303 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:47.150760 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 06:50:47.150891 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:50:47.152045 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:47.138665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:34189 (local address 127.14.116.194:35239)
0619 06:50:47.139117 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:47.139167 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:47.140295 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:47.141103 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:47.141115 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:47.141466 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:47.142175 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:47.142198 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.143340 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.143343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:47.144374 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:47.144384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.144892 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.144896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:47.145109 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:47.145732 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:47.145748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:47.147143 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:47.149104 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:47.149110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:47.149124 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:47.149380 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:47.149717 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:47.149721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:47.149723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:47.149877 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:47.150275 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:47.150281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:47.150529 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 06:50:47.150709 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 06:50:47.151012 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":142,"threads_started":1}
I20260619 06:50:47.153287 27937 heartbeater.cc:347] Connected to a master server at 127.14.116.254:34189
I20260619 06:50:47.153537 27937 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:47.154083 27937 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
W20260619 06:50:47.155910 27937 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:34189 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:50:47.156019 27937 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:47.156180 27937 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
I20260619 06:50:47.156983 27937 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:47.157145 27937 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
W20260619 06:50:47.157902 27937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:50:47.270424 27940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:47.270714 27940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:47.270787 27940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:47.275833 27940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:47.275954 27940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:47.276057 27940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:50:47.283015 27940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.116.212:35947/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:40249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=41869
--tserver_master_addrs=127.14.116.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:47.284531 27940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:47.285880 27940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:47.295264 27947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:47.295500 27949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:47.295282 27946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:47.295862 27940 server_base.cc:1034] running on GCE node
I20260619 06:50:47.296218 27940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:47.296931 27940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:47.304322 27940 hybrid_clock.cc:648] HybridClock initialized: now 1781851847304283 us; error 47 us; skew 500 ppm
Jun 19 06:50:47 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:47.307348 27940 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:50:47.308423 27940 webserver.cc:460] Webserver started at http://127.14.116.195:41869/ using document root <none> and password file <none>
I20260619 06:50:47.308975 27940 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:47.309078 27940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:50:47 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:47.329339 27940 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260619 06:50:47.336292 27957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:47.337779 27940 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260619 06:50:47.338078 27940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9"
format_stamp: "Formatted at 2026-06-19 06:50:46 on dist-test-slave-9x3b"
server_key: "921cfafb5f3c0580b46208e473550b04"
server_key_iv: "42395bda7120c70160dd16d348e280e1"
server_key_version: "kuduclusterkey@0"
I20260619 06:50:47.338653 27940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:47.367707 27940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:47.370827 27940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:47.371096 27940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:47.371887 27940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:47.372910 27940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:47.373119 27940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:47.373219 27940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:47.373288 27940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:47.384056 27940 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:40249
I20260619 06:50:47.385368 27940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:50:47.386510 28070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:40249 every 8 connection(s)
I20260619 06:50:47.393752 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 27940
Jun 19 06:50:47 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:47.402657 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:47.389643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:59027 (local address 127.14.116.254:34189)
0619 06:50:47.389798 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:47.389812 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:47.390471 (+ 659us) server_negotiation.cc:407] Connection header received
0619 06:50:47.391716 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:47.391720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:47.391785 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:47.391842 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:47.393440 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.394078 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.395486 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.395841 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.395904 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:47.398667 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:47.398670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:47.398673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:47.398701 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:47.400227 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:47.400826 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:47.400832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:47.400834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:47.400910 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:47.401315 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:47.401319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:47.401321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:47.401538 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:47.401644 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:47.402327 (+ 683us) server_negotiation.cc:299] Negotiation successful
0619 06:50:47.402453 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:50:47.403434 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:47.388547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:34189 (local address 127.14.116.195:59027)
0619 06:50:47.390309 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:47.390364 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:47.391439 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:47.392250 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:47.392260 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:47.392612 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:47.393260 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:47.393279 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.394234 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.394237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:47.395360 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:47.395371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.396058 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.396061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:47.396206 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:47.396883 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:47.396902 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:47.398503 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:47.400344 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:47.400349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:47.400365 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:47.400714 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:47.401027 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:47.401030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:47.401032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:47.401186 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:47.401663 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:47.401670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:47.402029 (+ 359us) client_negotiation.cc:769] Sending connection context
0619 06:50:47.402252 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 06:50:47.402540 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"thread_start_us":139,"threads_started":1}
I20260619 06:50:47.404955 28071 heartbeater.cc:347] Connected to a master server at 127.14.116.254:34189
I20260619 06:50:47.405223 28071 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:47.405805 28071 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
W20260619 06:50:47.407516 28071 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:34189 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:50:47.407676 28071 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:47.407859 28071 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
I20260619 06:50:47.408711 28071 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:47.408900 28071 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
W20260619 06:50:47.409658 28071 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:50:47.410076 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:47.394486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.14.116.254:34189)
0619 06:50:47.394873 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:47.394882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:47.395184 (+ 302us) server_negotiation.cc:407] Connection header received
0619 06:50:47.395294 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:47.395297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:47.395344 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:47.395447 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:47.396396 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.397132 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.399127 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:47.400506 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:47.400613 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:47.403818 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:47.403821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:47.403823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:47.403842 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:47.406485 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:47.406991 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:47.406997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:47.407001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:47.407088 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:47.408999 (+ 1911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:47.409002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:47.409007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:47.409225 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:47.409463 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:47.409855 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 06:50:47.409956 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":67,"threads_started":1}
Jun 19 06:50:47 dist-test-slave-9x3b krb5kdc[26609](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781851847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:47.917927 27803 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:47.918229 27803 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
I20260619 06:50:47.919677 27604 ts_manager.cc:194] Registered new tserver with Master: c88d62afe7cb42e5b6d71ca6e39f668f (127.14.116.193:38115)
I20260619 06:50:47.921432 27604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:38517
Jun 19 06:50:48 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:48.158149 27937 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:48.158455 27937 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
I20260619 06:50:48.159215 27604 ts_manager.cc:194] Registered new tserver with Master: 9240cb0ab05a4631a6b2c517be71a0f7 (127.14.116.194:39077)
I20260619 06:50:48.159997 27604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:35239
Jun 19 06:50:48 dist-test-slave-9x3b krb5kdc[26609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for HTTP/127.14.116.212@KRBTEST.COM
I20260619 06:50:48.409883 28071 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:48.410161 28071 heartbeater.cc:511] Master 127.14.116.254:34189 requested a full tablet report, sending...
I20260619 06:50:48.410929 27604 ts_manager.cc:194] Registered new tserver with Master: c9861ed24c0d4df5a7a6cf8bc3dd15a9 (127.14.116.195:40249)
I20260619 06:50:48.411747 27604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:59027
I20260619 06:50:48.414634 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:50:48.433034 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.416343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.14.116.254:34189)
0619 06:50:48.416902 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:48.416907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:48.417228 (+ 321us) server_negotiation.cc:407] Connection header received
0619 06:50:48.417267 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:48.417270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:48.417315 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:48.417401 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:48.418334 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.420022 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.422189 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.423038 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.423140 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.425269 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:48.425273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:48.425276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:48.425301 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:48.427339 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:48.428039 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:48.428043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:48.428045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:48.428142 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:48.431677 (+ 3535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:48.431681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:48.431683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:48.431916 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:48.432053 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:48.432737 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 06:50:48.432869 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":106,"threads_started":1}
I20260619 06:50:48.437276 27604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:50:48.441859 27604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:50:48.498744 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.481318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:38115 (local address 127.0.0.1:51756)
0619 06:50:48.487519 (+ 6201us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:48.487581 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:48.487731 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:48.492517 (+ 4786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:48.492522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:48.492540 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:48.492710 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.492717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.494947 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.494950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.496338 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.496359 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.498189 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.498193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.498299 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.498340 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:50:48.498468 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 06:50:48.498584 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6010,"thread_start_us":118,"threads_started":1}
I20260619 06:50:48.501245 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.481463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.14.116.193:38115)
0619 06:50:48.491815 (+ 10352us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:48.491823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:48.491849 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:50:48.491931 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:48.491937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:48.492167 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:48.492342 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:48.493628 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.494813 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.497021 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.497857 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.497990 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.498123 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:48.500973 (+ 2850us) server_negotiation.cc:299] Negotiation successful
0619 06:50:48.501078 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10235,"thread_start_us":68,"threads_started":1}
I20260619 06:50:48.502681 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.483284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:39077 (local address 127.0.0.1:54862)
0619 06:50:48.488567 (+ 5283us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:48.488582 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:48.488667 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:48.494269 (+ 5602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:48.494273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:48.494289 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:48.494377 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.494383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.499872 (+ 5489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.499876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.501075 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.501085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.502311 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.502314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.502401 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.502423 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:50:48.502491 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:50:48.502552 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5215,"thread_start_us":88,"threads_started":1}
I20260619 06:50:48.503360 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.483446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.14.116.194:39077)
0619 06:50:48.492984 (+ 9538us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:48.492991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:48.493015 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:50:48.493088 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:48.493093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:48.493290 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:48.493448 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:48.495687 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.496728 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.501221 (+ 4493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.502023 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.502137 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.502252 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:48.503133 (+ 881us) server_negotiation.cc:299] Negotiation successful
0619 06:50:48.503225 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9425,"thread_start_us":108,"threads_started":1}
I20260619 06:50:48.506632 27868 tablet_service.cc:1467] Processing CreateTablet for tablet e43364848a9f4555ac9ab7b7ce80921c (DEFAULT_TABLE table=test-table [id=cd9120aacb9d4c9a8b43202fed494111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:48.508127 27725 tablet_service.cc:1467] Processing CreateTablet for tablet e43364848a9f4555ac9ab7b7ce80921c (DEFAULT_TABLE table=test-table [id=cd9120aacb9d4c9a8b43202fed494111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:48.508625 27868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43364848a9f4555ac9ab7b7ce80921c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:48.508757 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.483620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:40249 (local address 127.0.0.1:59546)
0619 06:50:48.490777 (+ 7157us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:48.490793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:48.490896 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:48.499111 (+ 8215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:48.499114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:48.499133 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:48.499234 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.499241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.506227 (+ 6986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.506231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.507283 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.507293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.508392 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.508397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.508499 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.508523 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:50:48.508573 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 06:50:48.508634 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5160,"thread_start_us":68,"threads_started":1}
I20260619 06:50:48.509142 27725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43364848a9f4555ac9ab7b7ce80921c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:48.509405 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.483785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.14.116.195:40249)
0619 06:50:48.496608 (+ 12823us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:48.496614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:48.496634 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:50:48.496699 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:48.496703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:48.496889 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:48.499398 (+ 2509us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:48.499539 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.505289 (+ 5750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.507454 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.508252 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.509010 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.509132 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:48.509178 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 06:50:48.509262 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12724,"thread_start_us":112,"threads_started":1}
I20260619 06:50:48.512161 27987 tablet_service.cc:1467] Processing CreateTablet for tablet e43364848a9f4555ac9ab7b7ce80921c (DEFAULT_TABLE table=test-table [id=cd9120aacb9d4c9a8b43202fed494111]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:48.513079 27987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43364848a9f4555ac9ab7b7ce80921c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:48.529853 28107 tablet_bootstrap.cc:492] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: Bootstrap starting.
I20260619 06:50:48.532292 28109 tablet_bootstrap.cc:492] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f: Bootstrap starting.
I20260619 06:50:48.532346 28107 tablet_bootstrap.cc:654] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:48.533195 28107 log.cc:824] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:48.534530 28109 tablet_bootstrap.cc:654] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:48.534953 28108 tablet_bootstrap.cc:492] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: Bootstrap starting.
I20260619 06:50:48.537075 28108 tablet_bootstrap.cc:654] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:48.537987 28109 log.cc:824] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:48.538932 28108 log.cc:824] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:48.540512 28107 tablet_bootstrap.cc:492] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: No bootstrap required, opened a new log
I20260619 06:50:48.541498 28107 ts_tablet_manager.cc:1397] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260619 06:50:48.541776 28109 tablet_bootstrap.cc:492] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f: No bootstrap required, opened a new log
I20260619 06:50:48.542093 28109 ts_tablet_manager.cc:1397] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 06:50:48.545562 28107 raft_consensus.cc:348] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } }
I20260619 06:50:48.545892 28107 raft_consensus.cc:374] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:48.545946 28107 raft_consensus.cc:729] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9240cb0ab05a4631a6b2c517be71a0f7, State: Initialized, Role: FOLLOWER
I20260619 06:50:48.547030 28109 raft_consensus.cc:348] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } }
I20260619 06:50:48.547595 28109 raft_consensus.cc:374] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:48.547737 28109 raft_consensus.cc:729] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88d62afe7cb42e5b6d71ca6e39f668f, State: Initialized, Role: FOLLOWER
I20260619 06:50:48.550544 28108 tablet_bootstrap.cc:492] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: No bootstrap required, opened a new log
I20260619 06:50:48.550800 28108 ts_tablet_manager.cc:1397] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20260619 06:50:48.554903 28108 raft_consensus.cc:348] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } }
I20260619 06:50:48.555666 28108 raft_consensus.cc:374] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:48.555778 28108 raft_consensus.cc:729] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9861ed24c0d4df5a7a6cf8bc3dd15a9, State: Initialized, Role: FOLLOWER
I20260619 06:50:48.555693 28107 consensus_queue.cc:260] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } }
I20260619 06:50:48.555483 28109 consensus_queue.cc:260] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } }
I20260619 06:50:48.556478 28108 consensus_queue.cc:260] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } }
I20260619 06:50:48.556883 28107 ts_tablet_manager.cc:1428] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260619 06:50:48.557257 27937 heartbeater.cc:503] Master 127.14.116.254:34189 was elected leader, sending a full tablet report...
I20260619 06:50:48.558118 28108 ts_tablet_manager.cc:1428] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:50:48.558512 28109 ts_tablet_manager.cc:1428] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f: Time spent starting tablet: real 0.016s user 0.002s sys 0.004s
I20260619 06:50:48.558917 27803 heartbeater.cc:503] Master 127.14.116.254:34189 was elected leader, sending a full tablet report...
I20260619 06:50:48.561475 28071 heartbeater.cc:503] Master 127.14.116.254:34189 was elected leader, sending a full tablet report...
I20260619 06:50:48.607579 28115 raft_consensus.cc:482] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:50:48.607813 28115 raft_consensus.cc:504] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } }
I20260619 06:50:48.609112 28115 leader_election.cc:288] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c88d62afe7cb42e5b6d71ca6e39f668f (127.14.116.193:38115), 9240cb0ab05a4631a6b2c517be71a0f7 (127.14.116.194:39077)
I20260619 06:50:48.613680 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.609525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:48653 (local address 127.14.116.194:39077)
0619 06:50:48.609653 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:48.609660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:48.609929 (+ 269us) server_negotiation.cc:407] Connection header received
0619 06:50:48.610083 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:48.610086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:48.610136 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:48.610216 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:48.610506 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.611274 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.612711 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.613093 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.613222 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.613279 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:48.613455 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 06:50:48.613542 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:50:48.613798 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.609473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:52919 (local address 127.14.116.193:38115)
0619 06:50:48.609595 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:48.609601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:48.609759 (+ 158us) server_negotiation.cc:407] Connection header received
0619 06:50:48.609972 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:48.609975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:48.610023 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:48.610069 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:48.610477 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.611051 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.612680 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.613276 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.613396 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.613443 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:48.613606 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 06:50:48.613681 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 06:50:48.613878 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.609332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:38115 (local address 127.14.116.195:52919)
0619 06:50:48.609675 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:48.609689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:48.609836 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:48.610199 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:48.610202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:48.610217 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:48.610357 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.610362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.611157 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.611163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.612548 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.612553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.613379 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.613385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.613488 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.613507 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:50:48.613575 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:50:48.613641 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260619 06:50:48.614064 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.609332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:39077 (local address 127.14.116.195:48653)
0619 06:50:48.609827 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:48.609839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:48.609970 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:48.610226 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:48.610229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:48.610243 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:48.610384 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.610390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.611381 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.611383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.612542 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:48.612553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.613213 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.613217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:48.613312 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.613337 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:50:48.613834 (+ 497us) client_negotiation.cc:240] Negotiation successful
0619 06:50:48.613916 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":236,"thread_start_us":99,"threads_started":1}
I20260619 06:50:48.614550 27757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43364848a9f4555ac9ab7b7ce80921c" candidate_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" is_pre_election: true
I20260619 06:50:48.614640 27880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43364848a9f4555ac9ab7b7ce80921c" candidate_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" is_pre_election: true
I20260619 06:50:48.614880 27757 raft_consensus.cc:2454] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9861ed24c0d4df5a7a6cf8bc3dd15a9 in term 0.
I20260619 06:50:48.615002 27880 raft_consensus.cc:2454] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9861ed24c0d4df5a7a6cf8bc3dd15a9 in term 0.
I20260619 06:50:48.615358 27958 leader_election.cc:302] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c88d62afe7cb42e5b6d71ca6e39f668f, c9861ed24c0d4df5a7a6cf8bc3dd15a9; no voters:
I20260619 06:50:48.615672 28115 raft_consensus.cc:2781] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:50:48.615758 28115 raft_consensus.cc:482] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:50:48.615809 28115 raft_consensus.cc:3037] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:48.616911 28115 raft_consensus.cc:504] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } }
I20260619 06:50:48.617293 28115 leader_election.cc:288] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [CANDIDATE]: Term 1 election: Requested vote from peers c88d62afe7cb42e5b6d71ca6e39f668f (127.14.116.193:38115), 9240cb0ab05a4631a6b2c517be71a0f7 (127.14.116.194:39077)
I20260619 06:50:48.617604 27757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43364848a9f4555ac9ab7b7ce80921c" candidate_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f"
I20260619 06:50:48.617689 27880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43364848a9f4555ac9ab7b7ce80921c" candidate_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9240cb0ab05a4631a6b2c517be71a0f7"
I20260619 06:50:48.617714 27757 raft_consensus.cc:3037] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:48.617794 27880 raft_consensus.cc:3037] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:48.618637 27757 raft_consensus.cc:2454] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9861ed24c0d4df5a7a6cf8bc3dd15a9 in term 1.
I20260619 06:50:48.618896 27958 leader_election.cc:302] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c88d62afe7cb42e5b6d71ca6e39f668f, c9861ed24c0d4df5a7a6cf8bc3dd15a9; no voters:
I20260619 06:50:48.619050 28115 raft_consensus.cc:2781] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:50:48.619246 28115 raft_consensus.cc:686] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 1 LEADER]: Becoming Leader. State: Replica: c9861ed24c0d4df5a7a6cf8bc3dd15a9, State: Running, Role: LEADER
I20260619 06:50:48.619567 28115 consensus_queue.cc:237] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } }
I20260619 06:50:48.619721 27880 raft_consensus.cc:2454] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9861ed24c0d4df5a7a6cf8bc3dd15a9 in term 1.
I20260619 06:50:48.623085 27603 catalog_manager.cc:5495] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9861ed24c0d4df5a7a6cf8bc3dd15a9 (127.14.116.195). New cstate: current_term: 1 leader_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9861ed24c0d4df5a7a6cf8bc3dd15a9" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 40249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 } health_report { overall_health: UNKNOWN } } }
W20260619 06:50:48.637463 27804 tablet.cc:2369] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:50:48.638087 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.633806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.14.116.195:40249)
0619 06:50:48.633945 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:48.633950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:48.633965 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:50:48.634059 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:48.634062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:48.634119 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:48.634235 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:50:48.634432 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.635189 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.636057 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:48.636392 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:48.636488 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:48.636829 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:50:48.637356 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:50:48.637464 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:48.637887 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 06:50:48.637943 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 06:50:48.641552 28072 tablet.cc:2369] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:50:48.641661 27938 tablet.cc:2369] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:50:48.645005 27880 raft_consensus.cc:1261] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [term 1 FOLLOWER]: Refusing update from remote peer c9861ed24c0d4df5a7a6cf8bc3dd15a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:48.645108 27757 raft_consensus.cc:1261] T e43364848a9f4555ac9ab7b7ce80921c P c88d62afe7cb42e5b6d71ca6e39f668f [term 1 FOLLOWER]: Refusing update from remote peer c9861ed24c0d4df5a7a6cf8bc3dd15a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:48.645735 28115 consensus_queue.cc:1035] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c88d62afe7cb42e5b6d71ca6e39f668f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:48.645905 28125 consensus_queue.cc:1035] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9240cb0ab05a4631a6b2c517be71a0f7" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:48.655577 28129 mvcc.cc:205] Tried to move back new op lower bound from 7298465172043087872 to 7298465171948613632. Current Snapshot: MvccSnapshot[applied={T|T < 7298465172043087872}]
I20260619 06:50:48.662384 28130 mvcc.cc:205] Tried to move back new op lower bound from 7298465172043087872 to 7298465171948613632. Current Snapshot: MvccSnapshot[applied={T|T < 7298465172043087872}]
I20260619 06:50:48.663142 27604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:50:48.663303 27604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:50:48.665786 27604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b04935251cfc4801b1c68d4e2cc24c82: Sending DeleteTablet for 3 replicas of tablet e43364848a9f4555ac9ab7b7ce80921c
I20260619 06:50:48.666625 27868 tablet_service.cc:1514] Processing DeleteTablet for tablet e43364848a9f4555ac9ab7b7ce80921c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:48 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:54862
I20260619 06:50:48.666631 27725 tablet_service.cc:1514] Processing DeleteTablet for tablet e43364848a9f4555ac9ab7b7ce80921c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:48 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:51756
I20260619 06:50:48.667088 27987 tablet_service.cc:1514] Processing DeleteTablet for tablet e43364848a9f4555ac9ab7b7ce80921c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:48 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:59546
I20260619 06:50:48.667744 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 27643
I20260619 06:50:48.671722 28137 tablet_replica.cc:330] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: stopping tablet replica
I20260619 06:50:48.672047 28137 raft_consensus.cc:2229] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:50:48.672363 28137 raft_consensus.cc:2258] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:48.674504 28137 ts_tablet_manager.cc:1905] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:48.674808 28139 tablet_replica.cc:330] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: stopping tablet replica
I20260619 06:50:48.675199 28139 raft_consensus.cc:2229] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:50:48.676295 28139 raft_consensus.cc:2258] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:48.677238 28137 ts_tablet_manager.cc:1918] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:48.677330 28137 log.cc:1196] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/e43364848a9f4555ac9ab7b7ce80921c
I20260619 06:50:48.677654 28137 ts_tablet_manager.cc:1939] T e43364848a9f4555ac9ab7b7ce80921c P 9240cb0ab05a4631a6b2c517be71a0f7: Deleting consensus metadata
I20260619 06:50:48.678294 28139 ts_tablet_manager.cc:1905] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:48.678946 27591 catalog_manager.cc:4841] TS 9240cb0ab05a4631a6b2c517be71a0f7 (127.14.116.194:39077): tablet e43364848a9f4555ac9ab7b7ce80921c (table test-table [id=cd9120aacb9d4c9a8b43202fed494111]) successfully deleted
W20260619 06:50:48.680685 27589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:38115 (error 108)
I20260619 06:50:48.681022 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 27805
I20260619 06:50:48.682397 28139 ts_tablet_manager.cc:1918] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:48.682550 28139 log.cc:1196] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/e43364848a9f4555ac9ab7b7ce80921c
I20260619 06:50:48.682922 28139 ts_tablet_manager.cc:1939] T e43364848a9f4555ac9ab7b7ce80921c P c9861ed24c0d4df5a7a6cf8bc3dd15a9: Deleting consensus metadata
I20260619 06:50:48.684092 27589 catalog_manager.cc:4841] TS c9861ed24c0d4df5a7a6cf8bc3dd15a9 (127.14.116.195:40249): tablet e43364848a9f4555ac9ab7b7ce80921c (table test-table [id=cd9120aacb9d4c9a8b43202fed494111]) successfully deleted
I20260619 06:50:48.685027 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:48.684756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:38115 (local address 127.0.0.1:51758)
0619 06:50:48.684871 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:48.684940 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:38115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 06:50:48.685214 27589 catalog_manager.cc:4584] TS c88d62afe7cb42e5b6d71ca6e39f668f (127.14.116.193:38115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e43364848a9f4555ac9ab7b7ce80921c: Network error: Client connection negotiation failed: client connection to 127.14.116.193:38115: connect: Connection refused (error 111)
I20260619 06:50:48.688166 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 27940
I20260619 06:50:48.695140 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 27572
I20260619 06:50:48.703802 14803 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:50:49.039228 14803 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:50:49.039538 14803 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:50:49.384079 14803 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:50:49.384 UTC [26639] LOG: received smart shutdown request
2026-06-19 06:50:49.387 UTC [26639] LOG: background worker "logical replication launcher" (PID 26648) exited with exit code 1
2026-06-19 06:50:49.388 UTC [26643] LOG: shutting down
2026-06-19 06:50:49.530 UTC [26639] LOG: database system is shut down
2026-06-19T06:50:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46230 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:50:49 dist-test-slave-9x3b krb5kdc[28149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:50:49 dist-test-slave-9x3b krb5kdc[28149](info): set up 2 sockets
Jun 19 06:50:49 dist-test-slave-9x3b krb5kdc[28149](info): commencing operation
krb5kdc: starting...
W20260619 06:50:50.703661 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:50:50 dist-test-slave-9x3b krb5kdc[28149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:50:50.821924 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:40623
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40623,127.14.116.253:37805,127.14.116.252:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:50.927783 28162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:50.928053 28162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:50.928138 28162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:50.932077 28162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:50.932147 28162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:50.932184 28162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:50.932241 28162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:50.932281 28162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:50.937254 28162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40623,127.14.116.253:37805,127.14.116.252:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:50.938196 28162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:50.939119 28162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:50.947019 28168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:50.947048 28170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:50.947048 28167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:50.947156 28162 server_base.cc:1034] running on GCE node
I20260619 06:50:50.947803 28162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:50.948724 28162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:50.949874 28162 hybrid_clock.cc:648] HybridClock initialized: now 1781851850949845 us; error 41 us; skew 500 ppm
Jun 19 06:50:50 dist-test-slave-9x3b krb5kdc[28149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851850, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:50.958673 28162 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:50:50.959563 28162 webserver.cc:460] Webserver started at http://127.14.116.254:35805/ using document root <none> and password file <none>
I20260619 06:50:50.960105 28162 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:50.960165 28162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:50.960326 28162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:50:50.962198 28162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce"
format_stamp: "Formatted at 2026-06-19 06:50:50 on dist-test-slave-9x3b"
server_key: "fe1c51f02f0bee5f0f3b3cb0a0269395"
server_key_iv: "119c235627f632b36985be4dc90dacfb"
server_key_version: "encryptionkey@0"
I20260619 06:50:50.962694 28162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce"
format_stamp: "Formatted at 2026-06-19 06:50:50 on dist-test-slave-9x3b"
server_key: "fe1c51f02f0bee5f0f3b3cb0a0269395"
server_key_iv: "119c235627f632b36985be4dc90dacfb"
server_key_version: "encryptionkey@0"
I20260619 06:50:50.966105 28162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:50:50.968310 28177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:50.969453 28162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:50:50.969588 28162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce"
format_stamp: "Formatted at 2026-06-19 06:50:50 on dist-test-slave-9x3b"
server_key: "fe1c51f02f0bee5f0f3b3cb0a0269395"
server_key_iv: "119c235627f632b36985be4dc90dacfb"
server_key_version: "encryptionkey@0"
I20260619 06:50:50.969697 28162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:50.984606 28162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:50.987426 28162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:50.987643 28162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:50.994242 28162 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:40623
I20260619 06:50:50.994520 28229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:40623 every 8 connection(s)
I20260619 06:50:50.995165 28162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:50:50.996855 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 28162
I20260619 06:50:50.997008 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:50:50.997294 14803 external_mini_cluster.cc:1428] Setting key d4367bda0521c4752511169a8a0cb9bf
I20260619 06:50:50.997948 28230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:51.001587 28230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.116.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.253@KRBTEST.COM" created.
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851850, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:51.016898 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.004469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.14.116.254:40623)
0619 06:50:51.004758 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.004766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.004800 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:50:51.005257 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.005261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.005484 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.005637 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.006623 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.007785 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.009015 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.009575 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.009700 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.012091 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.012100 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.012115 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.012148 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.014073 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.014528 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.014534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.014536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.014676 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.014978 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.014984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.014987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.015465 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.015643 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.015969 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.016125 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260619 06:50:51.016897 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.003571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40623 (local address 127.0.0.1:34610)
0619 06:50:51.004068 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.004118 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:51.004969 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:51.005661 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:51.005665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:51.005972 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:51.006270 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.006291 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.007903 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.007910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.008894 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.008903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.009705 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.009707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.009782 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.010339 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:51.010355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:51.011944 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:51.014180 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.014187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.014193 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.014417 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.014772 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.014774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.014775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.014879 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.015574 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:51.015578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:51.015741 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 06:50:51.015883 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 06:50:51.016216 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":90,"thread_start_us":123,"threads_started":1}
I20260619 06:50:51.019220 28230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:50:51.020001 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.019754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:37805 (local address 127.0.0.1:35376)
0619 06:50:51.019874 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.019949 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:50:51.020212 28180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
I20260619 06:50:51.021045 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.020794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:37805 (local address 127.0.0.1:35380)
0619 06:50:51.020959 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.020999 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 06:50:51.021173 28230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.253:37805: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab.
Entry for principal kudu/127.14.116.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab.
I20260619 06:50:51.022667 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.253:37805
--webserver_interface=127.14.116.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab
--principal=kudu/127.14.116.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40623,127.14.116.253:37805,127.14.116.252:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:50:51.070490 28230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } attempt: 1
I20260619 06:50:51.071439 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.071098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:37805 (local address 127.0.0.1:35388)
0619 06:50:51.071280 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.071349 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:50:51.072386 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.072163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:37805 (local address 127.0.0.1:35402)
0619 06:50:51.072273 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.072324 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 06:50:51.072515 28230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.253:37805: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
W20260619 06:50:51.131100 28237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:51.131368 28237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:51.131451 28237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:51.135380 28237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:51.135459 28237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:51.135499 28237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:51.135536 28237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:51.135571 28237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:51.140686 28237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40623,127.14.116.253:37805,127.14.116.252:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab
--principal=kudu/127.14.116.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.253:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.116.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:51.140663 28230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } attempt: 2
I20260619 06:50:51.141425 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.141138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:37805 (local address 127.0.0.1:35418)
0619 06:50:51.141292 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.141354 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 06:50:51.141749 28237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:51.142112 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.141931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:37805 (local address 127.0.0.1:35430)
0619 06:50:51.142015 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.142058 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 06:50:51.142261 28230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.253:37805: Network error: Client connection negotiation failed: client connection to 127.14.116.253:37805: connect: Connection refused (error 111)
I20260619 06:50:51.142846 28237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:51.150650 28242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:51.150784 28237 server_base.cc:1034] running on GCE node
W20260619 06:50:51.150830 28243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:51.150686 28245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:51.151379 28237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:51.152428 28237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:51.153618 28237 hybrid_clock.cc:648] HybridClock initialized: now 1781851851153601 us; error 28 us; skew 500 ppm
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851851, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:51.156224 28237 init.cc:377] Logged in from keytab as kudu/127.14.116.253@KRBTEST.COM (short username kudu)
I20260619 06:50:51.157043 28237 webserver.cc:460] Webserver started at http://127.14.116.253:46167/ using document root <none> and password file <none>
I20260619 06:50:51.157552 28237 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:51.157642 28237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:51.157867 28237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:50:51.159823 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/data/instance:
uuid: "44b680318b88499c9d72491777bf90a2"
format_stamp: "Formatted at 2026-06-19 06:50:51 on dist-test-slave-9x3b"
server_key: "77a3fb87bbfd75f7aa86b38907d46af1"
server_key_iv: "8fa48a231a5a4a3559756afeadc91c91"
server_key_version: "encryptionkey@0"
I20260619 06:50:51.160364 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/wal/instance:
uuid: "44b680318b88499c9d72491777bf90a2"
format_stamp: "Formatted at 2026-06-19 06:50:51 on dist-test-slave-9x3b"
server_key: "77a3fb87bbfd75f7aa86b38907d46af1"
server_key_iv: "8fa48a231a5a4a3559756afeadc91c91"
server_key_version: "encryptionkey@0"
I20260619 06:50:51.163751 28237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:50:51.165918 28252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:51.167011 28237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:50:51.167150 28237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/wal
uuid: "44b680318b88499c9d72491777bf90a2"
format_stamp: "Formatted at 2026-06-19 06:50:51 on dist-test-slave-9x3b"
server_key: "77a3fb87bbfd75f7aa86b38907d46af1"
server_key_iv: "8fa48a231a5a4a3559756afeadc91c91"
server_key_version: "encryptionkey@0"
I20260619 06:50:51.167253 28237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:51.190583 28237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:51.193734 28237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:51.193943 28237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:51.200500 28237 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.253:37805
I20260619 06:50:51.200769 28304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.253:37805 every 8 connection(s)
I20260619 06:50:51.201453 28237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
I20260619 06:50:51.204068 28305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:51.207512 28305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:50:51.208061 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 28237
I20260619 06:50:51.208189 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-1/wal/instance
I20260619 06:50:51.208472 14803 external_mini_cluster.cc:1428] Setting key 5d89d1ad91d75fdd80ac99a32dfe40db
WARNING: no policy specified for kudu/127.14.116.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.252@KRBTEST.COM" created.
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851851, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:51.223218 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.209585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.14.116.254:40623)
0619 06:50:51.209742 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.209747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.210632 (+ 885us) server_negotiation.cc:407] Connection header received
0619 06:50:51.211778 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.211782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.211838 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.211945 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.213113 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.213637 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.214768 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.215022 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.215158 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.219667 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.219671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.219674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.219700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.221133 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.221597 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.221604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.221607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.221688 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.222040 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.222045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.222047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.222292 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.222409 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.222914 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.223035 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:50:51.224011 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.209952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40623 (local address 127.0.0.1:34620)
0619 06:50:51.210444 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.210503 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:51.211533 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:51.212106 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:51.212113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:51.212512 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:51.212971 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.212983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.213774 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.213777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.214642 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.214651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.215191 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.215195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.215363 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.215989 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:51.216004 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:51.219454 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:51.221239 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.221243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.221253 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.221488 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.221789 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.221792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.221794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.221924 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.222418 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:51.222425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:51.222657 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 06:50:51.222849 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 06:50:51.223155 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260619 06:50:51.225410 28305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851851, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab.
Entry for principal kudu/127.14.116.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab.
I20260619 06:50:51.233451 28230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } attempt: 3
I20260619 06:50:51.234299 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.252:32915
--webserver_interface=127.14.116.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab
--principal=kudu/127.14.116.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40623,127.14.116.253:37805,127.14.116.252:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:50:51.235011 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.226008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.14.116.253:37805)
0619 06:50:51.226299 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.226313 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.226361 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:50:51.226489 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.226493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.226629 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.226758 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.227163 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.228065 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.228864 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.229195 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.229277 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.231279 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.231315 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.231334 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.231354 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.232820 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.233236 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.233242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.233244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.233366 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.234015 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.234021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.234026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.234312 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.234461 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.234730 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.234867 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":190,"thread_start_us":123,"threads_started":1}
I20260619 06:50:51.235065 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.225920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:37805 (local address 127.0.0.1:35438)
0619 06:50:51.226296 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.226311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:51.226392 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:51.226736 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:51.226739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:51.226916 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:51.226999 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.227004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.228216 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.228219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.228744 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.228752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.229292 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.229294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.229365 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.229808 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:51.229818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:51.231128 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:51.232914 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.232917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.232918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.233116 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.233496 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.233499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.233501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.233586 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.234423 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:51.234428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:51.234537 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 06:50:51.234657 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 06:50:51.234813 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326}
I20260619 06:50:51.237044 28305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:50:51.237671 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.237466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52788)
0619 06:50:51.237561 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.237622 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:50:51.237859 28253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
I20260619 06:50:51.238629 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.238437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52800)
0619 06:50:51.238538 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.238586 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 06:50:51.239121 28305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:32915: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851850, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
I20260619 06:50:51.243835 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.233925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.14.116.253:37805)
0619 06:50:51.235192 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.235198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.235213 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:50:51.235251 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.235253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.235306 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.235839 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.235921 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.236398 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.237938 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.238383 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.238899 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.240811 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.240814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.240821 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.240841 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.242185 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.242684 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.242686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.242687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.242750 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.243048 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.243050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.243051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.243185 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.243253 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.243585 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.243719 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1196,"thread_start_us":97,"threads_started":1}
I20260619 06:50:51.243968 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.233862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:37805 (local address 127.0.0.1:35444)
0619 06:50:51.233979 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.233995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:51.234069 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:51.235433 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:51.235436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:51.235597 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:51.235731 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.235738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.236506 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.236512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.237817 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.237828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.238521 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.238528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.238673 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.239142 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:51.239158 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:51.240684 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:51.242303 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.242309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.242313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.242589 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.242840 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.242846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.242851 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.242953 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.243286 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:51.243290 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:51.243408 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 06:50:51.243546 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 06:50:51.243728 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 06:50:51.244997 28230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:50:51.245627 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.245426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52810)
0619 06:50:51.245532 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.245574 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:50:51.246289 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.246079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52826)
0619 06:50:51.246207 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.246244 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 06:50:51.246469 28230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:32915: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
I20260619 06:50:51.277639 28230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } attempt: 1
I20260619 06:50:51.278383 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.278130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52832)
0619 06:50:51.278264 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.278317 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:50:51.279080 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.278906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52838)
0619 06:50:51.278999 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.279037 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:50:51.279242 28230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:32915: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
I20260619 06:50:51.288353 28305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } attempt: 1
I20260619 06:50:51.289072 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.288789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52844)
0619 06:50:51.288934 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.288995 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:50:51.289760 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.289548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52848)
0619 06:50:51.289656 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.289705 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:50:51.289916 28305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:32915: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
W20260619 06:50:51.345497 28312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:51.345721 28312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:51.345790 28312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:51.349745 28312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:51.349799 28312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:51.349818 28312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:51.349852 28312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:51.349867 28312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:51.354395 28230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } attempt: 2
I20260619 06:50:51.355201 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.354922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52856)
0619 06:50:51.355067 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.355127 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 06:50:51.355320 28312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40623,127.14.116.253:37805,127.14.116.252:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab
--principal=kudu/127.14.116.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.252:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.116.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:51.355981 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.355791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52870)
0619 06:50:51.355891 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.355933 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:50:51.356208 28230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:32915: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
I20260619 06:50:51.356559 28312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:51.357580 28312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:50:51.358083 28305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } attempt: 2
I20260619 06:50:51.358739 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.358470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52872)
0619 06:50:51.358607 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.358666 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 06:50:51.359460 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.359228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52878)
0619 06:50:51.359349 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.359401 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:50:51.359624 28305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:32915: Network error: Client connection negotiation failed: client connection to 127.14.116.252:32915: connect: Connection refused (error 111)
W20260619 06:50:51.365988 28319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:51.366024 28318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:51.366048 28312 server_base.cc:1034] running on GCE node
W20260619 06:50:51.365988 28321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:51.366796 28312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:51.367849 28312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:51.369006 28312 hybrid_clock.cc:648] HybridClock initialized: now 1781851851368992 us; error 22 us; skew 500 ppm
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851851, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:51.371651 28312 init.cc:377] Logged in from keytab as kudu/127.14.116.252@KRBTEST.COM (short username kudu)
I20260619 06:50:51.372514 28312 webserver.cc:460] Webserver started at http://127.14.116.252:33111/ using document root <none> and password file <none>
I20260619 06:50:51.373041 28312 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:51.373099 28312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:51.373253 28312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:50:51.375283 28312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/data/instance:
uuid: "73db467734c448b08fccbc7edb76d0c5"
format_stamp: "Formatted at 2026-06-19 06:50:51 on dist-test-slave-9x3b"
server_key: "b2593f15878d8ef95c87cdcf8f66df5b"
server_key_iv: "bf85f85297da4f8834f8ee98fc0504b7"
server_key_version: "encryptionkey@0"
I20260619 06:50:51.375831 28312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/wal/instance:
uuid: "73db467734c448b08fccbc7edb76d0c5"
format_stamp: "Formatted at 2026-06-19 06:50:51 on dist-test-slave-9x3b"
server_key: "b2593f15878d8ef95c87cdcf8f66df5b"
server_key_iv: "bf85f85297da4f8834f8ee98fc0504b7"
server_key_version: "encryptionkey@0"
I20260619 06:50:51.379189 28312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:50:51.381557 28328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:51.382690 28312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:50:51.382829 28312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/wal
uuid: "73db467734c448b08fccbc7edb76d0c5"
format_stamp: "Formatted at 2026-06-19 06:50:51 on dist-test-slave-9x3b"
server_key: "b2593f15878d8ef95c87cdcf8f66df5b"
server_key_iv: "bf85f85297da4f8834f8ee98fc0504b7"
server_key_version: "encryptionkey@0"
I20260619 06:50:51.382937 28312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:51.407295 28312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:51.409941 28312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:51.410144 28312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:51.416906 28312 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.252:32915
I20260619 06:50:51.417189 28380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.252:32915 every 8 connection(s)
I20260619 06:50:51.417825 28312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
I20260619 06:50:51.420472 28381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:51.421885 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 28312
I20260619 06:50:51.422025 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781851695409368-14803-0/minicluster-data/master-2/wal/instance
I20260619 06:50:51.422323 14803 external_mini_cluster.cc:1428] Setting key 9873153fada7a4d376ade7e5a54cf571
I20260619 06:50:51.424623 28381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851851, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:51.435827 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.426718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.14.116.254:40623)
0619 06:50:51.426866 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.426870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.427118 (+ 248us) server_negotiation.cc:407] Connection header received
0619 06:50:51.427240 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.427243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.427291 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.427374 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.428091 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.428575 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.429519 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.429840 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.429953 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.432664 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.432668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.432671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.432702 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.433981 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.434457 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.434460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.434461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.434533 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.434842 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.434844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.434845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.435021 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.435095 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.435497 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.435643 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:50:51.436728 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.423643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.14.116.252:32915)
0619 06:50:51.424030 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.424048 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.424088 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:50:51.424874 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.424897 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.425330 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.425673 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.426384 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.427789 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.428752 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.429220 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.429340 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.432116 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.432123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.432137 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.432169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.433964 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.434396 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.434403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.434405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.434519 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.434791 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.434795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.434798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.435182 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.435370 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.435587 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.435850 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":249,"thread_start_us":88,"threads_started":1}
I20260619 06:50:51.436726 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.426600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40623 (local address 127.0.0.1:34622)
0619 06:50:51.426997 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.427040 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:51.427150 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:51.427423 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:51.427427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:51.427797 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:51.427970 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.427978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.428713 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.428717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.429404 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.429414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.430009 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.430014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.430138 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.430593 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:51.430615 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:51.432525 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:51.434112 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.434118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.434125 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.434362 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.434639 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.434641 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.434642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.434734 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.435121 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:51.435127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:51.435252 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 06:50:51.435421 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 06:50:51.435957 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":154,"thread_start_us":135,"threads_started":1}
I20260619 06:50:51.437839 28381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851851, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:51.446049 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.438311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:37805 (local address 127.0.0.1:35452)
0619 06:50:51.438470 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.438482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:51.438547 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:51.438875 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:51.438877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:51.439036 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:51.439171 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.439179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.439916 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.439918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.440741 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.440750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.441298 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.441300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.441368 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.441747 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:51.441762 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:51.442972 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:51.444656 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.444658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.444659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.444832 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.445124 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.445128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.445130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.445229 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.445593 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:51.445596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:51.445695 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 06:50:51.445795 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:50:51.445943 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 06:50:51.446050 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.438458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.14.116.253:37805)
0619 06:50:51.438636 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.438640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.438655 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:50:51.438700 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.438705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.438758 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.438848 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.439263 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.439832 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.440855 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.441193 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.441292 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.443105 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.443108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.443111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.443140 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.444553 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.444941 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.444945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.444949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.445029 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.445326 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.445330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.445331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.445497 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.445602 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.445855 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.445945 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 06:50:51.447046 28381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:50:51.449011 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.438925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.14.116.254:40623)
0619 06:50:51.439032 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.439035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.439116 (+ 81us) server_negotiation.cc:407] Connection header received
0619 06:50:51.439931 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.439933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.439968 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.440020 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.440708 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.441396 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.442481 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.442691 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.442817 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.445456 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.445459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.445462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.445490 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.447307 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.447779 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.447785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.447790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.447877 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.448198 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.448201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.448203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.448429 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.448579 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.448796 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.448879 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 06:50:51.450773 28305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } attempt: 3
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851851, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
I20260619 06:50:51.455356 28230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } attempt: 3
I20260619 06:50:51.456146 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.447515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52892)
0619 06:50:51.447662 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.447676 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:51.447751 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:51.448007 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:51.448010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:51.448143 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:51.448249 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.448256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.449652 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.449659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.450398 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.450410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.451114 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.451117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.451184 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.451589 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:51.451602 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:51.452894 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:51.454688 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.454691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.454693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.454895 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.455160 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.455163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.455165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.455242 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.455677 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:51.455680 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:51.455761 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 06:50:51.455855 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:50:51.455974 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
I20260619 06:50:51.456579 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.447676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.14.116.252:32915)
0619 06:50:51.447760 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.447763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.447774 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:50:51.447842 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.447851 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.447883 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.447991 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.448365 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.449464 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.450510 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.450912 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.451009 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.453020 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.453023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.453026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.453046 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.454577 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.454987 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.454993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.454995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.455070 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.455344 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.455347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.455349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.455554 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.456320 (+ 766us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.456386 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.456483 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851851, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
I20260619 06:50:51.460391 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.451133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52902)
0619 06:50:51.451219 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.451234 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:51.451311 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:51.451924 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:51.451930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:51.452116 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:51.452187 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.452192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.453213 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.453219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.454069 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.454080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.454722 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.454729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.454838 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.455294 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:51.455308 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:51.456949 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:51.458840 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.458843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.458845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.459039 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.459317 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.459322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.459324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.459427 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.459901 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:51.459904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:51.460008 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 06:50:51.460087 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:50:51.460216 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 06:50:51.461014 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.451422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.14.116.252:32915)
0619 06:50:51.451690 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.451703 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.451715 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:50:51.451747 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.451749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.451797 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.451881 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.452297 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.453098 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.454208 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.454602 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.454694 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.457070 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.457074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.457079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.457106 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.458734 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.459137 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.459141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.459143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.459225 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.459526 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.459529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.459531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.459782 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.460673 (+ 891us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.460747 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.460857 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":115,"threads_started":1}
I20260619 06:50:51.461645 28381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5: Bootstrap starting.
Jun 19 06:50:51 dist-test-slave-9x3b krb5kdc[28149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851850, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
I20260619 06:50:51.463078 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.450093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.14.116.253:37805)
0619 06:50:51.450193 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.450197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.450211 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:50:51.450294 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.450296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.450332 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.450419 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.451095 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.452285 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.453467 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.453664 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.453813 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.457733 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.457736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.457739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.457761 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.460154 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.460644 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.460647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.460649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.460727 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.461034 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.461037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.461038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.462215 (+ 1177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.462400 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.462787 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.462913 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 06:50:51.463903 28305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2: Bootstrap starting.
I20260619 06:50:51.465186 28381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:51.465699 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.455716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:32915 (local address 127.0.0.1:52906)
0619 06:50:51.456764 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:51.456777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:51.456872 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:51.457139 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:51.457142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:51.457339 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:51.457435 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.457441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.458597 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.458601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.459345 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:51.459355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.460017 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.460020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:51.460113 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.460692 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:51.460707 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:51.462060 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:51.464148 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.464151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.464153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.464389 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.464683 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:51.464686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:51.464687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:51.464776 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:51.465213 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:51.465216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:51.465313 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 06:50:51.465433 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 06:50:51.465562 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975}
I20260619 06:50:51.465965 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.455915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.14.116.252:32915)
0619 06:50:51.456227 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.456236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.456826 (+ 590us) server_negotiation.cc:407] Connection header received
0619 06:50:51.456952 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.456954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.456986 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.457557 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.457658 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.458448 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.459470 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.459902 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.460308 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.462193 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.462196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.462199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.462221 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.464035 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.464496 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.464499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.464500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.464582 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.464885 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.464890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.464891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.465102 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.465198 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.465495 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.465598 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260619 06:50:51.466051 28381 log.cc:824] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:51.466331 28305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:51.467016 28305 log.cc:824] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:51.468659 28230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce: Bootstrap starting.
I20260619 06:50:51.469105 28305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2: No bootstrap required, opened a new log
I20260619 06:50:51.469105 28381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5: No bootstrap required, opened a new log
I20260619 06:50:51.470466 28230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:51.471186 28230 log.cc:824] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:51.473263 28230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce: No bootstrap required, opened a new log
I20260619 06:50:51.473459 28305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } }
I20260619 06:50:51.473739 28305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:51.473845 28305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b680318b88499c9d72491777bf90a2, State: Initialized, Role: FOLLOWER
I20260619 06:50:51.473718 28381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } }
I20260619 06:50:51.473943 28381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:51.474035 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:50:51.474051 28381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73db467734c448b08fccbc7edb76d0c5, State: Initialized, Role: FOLLOWER
I20260619 06:50:51.474498 28305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } }
I20260619 06:50:51.474498 28381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } }
I20260619 06:50:51.475791 28305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:50:51.475790 28381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:50:51.477480 28230 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } }
I20260619 06:50:51.477684 28230 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:51.477756 28230 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0db6cfef2e044a9b0ebb9ee6a9652ce, State: Initialized, Role: FOLLOWER
W20260619 06:50:51.480902 28418 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:50:51.481001 28418 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 06:50:51.480902 28421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:50:51.481132 28421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:50:51.479538 28312 master_runner.cc:186] Error getting master registration for 127.14.116.254:40623: OK, instance_id { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" instance_seqno: 1781851850992449 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:50:51.482937 28312 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:50:51.483042 28424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.483161 28424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:50:51.483244 28424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:50:51.479552 28237 master_runner.cc:186] Error getting master registration for 127.14.116.254:40623: OK, instance_id { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" instance_seqno: 1781851850992449 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:50:51.483374 28424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } }
I20260619 06:50:51.483462 28237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:50:51.483789 28230 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } }
I20260619 06:50:51.485183 28230 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:50:51.485426 28279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73db467734c448b08fccbc7edb76d0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b680318b88499c9d72491777bf90a2" is_pre_election: true
I20260619 06:50:51.485683 28279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73db467734c448b08fccbc7edb76d0c5 in term 0.
I20260619 06:50:51.485913 28204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73db467734c448b08fccbc7edb76d0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" is_pre_election: true
I20260619 06:50:51.486171 28204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73db467734c448b08fccbc7edb76d0c5 in term 0.
I20260619 06:50:51.486630 28332 leader_election.cc:302] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73db467734c448b08fccbc7edb76d0c5, d0db6cfef2e044a9b0ebb9ee6a9652ce; no voters:
I20260619 06:50:51.487061 28424 leader_election.cc:288] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0db6cfef2e044a9b0ebb9ee6a9652ce (127.14.116.254:40623), 44b680318b88499c9d72491777bf90a2 (127.14.116.253:37805)
I20260619 06:50:51.487293 28424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:50:51.487368 28424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:50:51.487422 28424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:51.488823 28424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } }
W20260619 06:50:51.489078 28183 tablet.cc:2369] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:50:51.489584 28204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73db467734c448b08fccbc7edb76d0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce"
I20260619 06:50:51.489698 28204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:51.490935 28204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73db467734c448b08fccbc7edb76d0c5 in term 1.
I20260619 06:50:51.492057 28332 leader_election.cc:302] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73db467734c448b08fccbc7edb76d0c5, d0db6cfef2e044a9b0ebb9ee6a9652ce; no voters:
I20260619 06:50:51.495100 28279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73db467734c448b08fccbc7edb76d0c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44b680318b88499c9d72491777bf90a2"
I20260619 06:50:51.495214 28279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:51.495492 28162 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260619 06:50:51.495595 28162 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:50:51.496457 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.476310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.14.116.254:40623)
0619 06:50:51.476424 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.476428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.483911 (+ 7483us) server_negotiation.cc:407] Connection header received
0619 06:50:51.484169 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.484172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.484217 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.484263 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.485563 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.486624 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.488513 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.488822 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.488913 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.491544 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.491548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.491551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.491575 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.493718 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.494281 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.494284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.494288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.494387 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.494999 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.495003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.495006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.495579 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.495712 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.496142 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.496259 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260619 06:50:51.496513 28279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73db467734c448b08fccbc7edb76d0c5 in term 1.
I20260619 06:50:51.496814 28424 leader_election.cc:288] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [CANDIDATE]: Term 1 election: Requested vote from peers d0db6cfef2e044a9b0ebb9ee6a9652ce (127.14.116.254:40623), 44b680318b88499c9d72491777bf90a2 (127.14.116.253:37805)
I20260619 06:50:51.496960 28424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:50:51.497038 28424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 73db467734c448b08fccbc7edb76d0c5, State: Running, Role: LEADER
I20260619 06:50:51.497340 28424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } }
I20260619 06:50:51.497932 28392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.498298 28392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:50:51.498660 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.476971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.14.116.253:37805)
0619 06:50:51.477067 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.477073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.488061 (+ 10988us) server_negotiation.cc:407] Connection header received
0619 06:50:51.488099 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.488102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.488143 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.488211 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.490193 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.490989 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.492154 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.492538 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.492626 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.494405 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.494411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.494416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.494447 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.496191 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.496651 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.496658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.496660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.496742 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.497050 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.497053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.497056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.497274 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.497367 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.498386 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.498501 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 06:50:51.499073 28391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73db467734c448b08fccbc7edb76d0c5. Latest consensus state: current_term: 1 leader_uuid: "73db467734c448b08fccbc7edb76d0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.499290 28391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:51.500867 28438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:50:51.501835 28438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 06:50:51.502138 28436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:50:51.502218 28436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:50:51.503875 28425 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.503993 28425 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:50:51.507552 28279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [term 1 FOLLOWER]: Refusing update from remote peer 73db467734c448b08fccbc7edb76d0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:51.508775 28391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:51.511085 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:51.477554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.14.116.252:32915)
0619 06:50:51.477721 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:51.477730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:51.497690 (+ 19960us) server_negotiation.cc:407] Connection header received
0619 06:50:51.497805 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:51.497809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:51.497854 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:51.497932 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:51.503738 (+ 5806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.504522 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.505609 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:51.505969 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:51.506047 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:51.507712 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:51.507716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:51.507719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:51.507760 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:51.509370 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.509849 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.509852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.509854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.509931 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:51.510194 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:51.510197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:51.510199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:51.510397 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:51.510506 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:51.510833 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 06:50:51.510942 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 06:50:51.512452 28392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73db467734c448b08fccbc7edb76d0c5. Latest consensus state: current_term: 1 leader_uuid: "73db467734c448b08fccbc7edb76d0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.512594 28392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:50:51.514344 28204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [term 1 FOLLOWER]: Refusing update from remote peer 73db467734c448b08fccbc7edb76d0c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:51.514897 28391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:51.518455 28391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73db467734c448b08fccbc7edb76d0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.518554 28391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:51.520255 28392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73db467734c448b08fccbc7edb76d0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.520385 28392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:50:51.520727 28425 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73db467734c448b08fccbc7edb76d0c5. Latest consensus state: current_term: 1 leader_uuid: "73db467734c448b08fccbc7edb76d0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.520856 28425 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:50:51.520816 28440 mvcc.cc:205] Tried to move back new op lower bound from 7298465183766671360 to 7298465183736139776. Current Snapshot: MvccSnapshot[applied={T|T < 7298465183766671360}]
I20260619 06:50:51.521341 28441 mvcc.cc:205] Tried to move back new op lower bound from 7298465183766671360 to 7298465183736139776. Current Snapshot: MvccSnapshot[applied={T|T < 7298465183766671360 or (T in {7298465183766671360})}]
I20260619 06:50:51.522703 28438 catalog_manager.cc:1347] Generated new cluster ID: d1d196847f474c16a1b2d595ac57be3e
I20260619 06:50:51.522763 28438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:50:51.529376 28424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73db467734c448b08fccbc7edb76d0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.529481 28424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:51.530530 28444 mvcc.cc:205] Tried to move back new op lower bound from 7298465183766671360 to 7298465183736139776. Current Snapshot: MvccSnapshot[applied={T|T < 7298465183766671360}]
I20260619 06:50:51.530659 28425 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73db467734c448b08fccbc7edb76d0c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0db6cfef2e044a9b0ebb9ee6a9652ce" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40623 } } peers { permanent_uuid: "44b680318b88499c9d72491777bf90a2" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 37805 } } peers { permanent_uuid: "73db467734c448b08fccbc7edb76d0c5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 32915 } } }
I20260619 06:50:51.530742 28425 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:50:51.551158 28438 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:50:51.552410 28438 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:50:51.575496 28438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73db467734c448b08fccbc7edb76d0c5: Generated new TSK 0
I20260619 06:50:51.576704 28438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:50:52.482352 28421 catalog_manager.cc:1259] Loaded cluster ID: d1d196847f474c16a1b2d595ac57be3e
I20260619 06:50:52.482443 28421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2: loading cluster ID for follower catalog manager: success
I20260619 06:50:52.484220 28421 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2: acquiring CA information for follower catalog manager: success
I20260619 06:50:52.485513 28421 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44b680318b88499c9d72491777bf90a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:50:52.503265 28436 catalog_manager.cc:1259] Loaded cluster ID: d1d196847f474c16a1b2d595ac57be3e
I20260619 06:50:52.503327 28436 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce: loading cluster ID for follower catalog manager: success
I20260619 06:50:52.505030 28436 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce: acquiring CA information for follower catalog manager: success
I20260619 06:50:52.506102 28436 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0db6cfef2e044a9b0ebb9ee6a9652ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:50:52.642742 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 28162
I20260619 06:50:52.649479 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 28237
I20260619 06:50:52.654986 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 28312
2026-06-19T06:50:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3004 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:50:52 dist-test-slave-9x3b krb5kdc[28455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:50:52 dist-test-slave-9x3b krb5kdc[28455](info): set up 2 sockets
Jun 19 06:50:52 dist-test-slave-9x3b krb5kdc[28455](info): commencing operation
krb5kdc: starting...
W20260619 06:50:53.696144 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:50:53 dist-test-slave-9x3b krb5kdc[28455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:50:53.810200 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:44369
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:53.914415 28468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:53.914642 28468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:53.914680 28468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:53.918473 28468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:53.918519 28468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:53.918537 28468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:53.918553 28468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:53.918568 28468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:53.923751 28468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:44369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:53.924598 28468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:53.925446 28468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:53.933104 28476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:53.933068 28473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:53.933140 28474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:53.933183 28468 server_base.cc:1034] running on GCE node
I20260619 06:50:53.933840 28468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:53.934813 28468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:53.935966 28468 hybrid_clock.cc:648] HybridClock initialized: now 1781851853935940 us; error 38 us; skew 500 ppm
Jun 19 06:50:53 dist-test-slave-9x3b krb5kdc[28455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851853, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:53.938431 28468 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:50:53.939297 28468 webserver.cc:460] Webserver started at http://127.14.116.254:35783/ using document root <none> and password file <none>
I20260619 06:50:53.939893 28468 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:53.939980 28468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:53.940196 28468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:50:53.942078 28468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "37d50280f05e4646ab8bae3d97c2267d"
format_stamp: "Formatted at 2026-06-19 06:50:53 on dist-test-slave-9x3b"
server_key: "5b843e2a7a9521fb159e998b08ebf236"
server_key_iv: "4bceff59001d8a0b8afc6af1cc425997"
server_key_version: "encryptionkey@0"
I20260619 06:50:53.942612 28468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "37d50280f05e4646ab8bae3d97c2267d"
format_stamp: "Formatted at 2026-06-19 06:50:53 on dist-test-slave-9x3b"
server_key: "5b843e2a7a9521fb159e998b08ebf236"
server_key_iv: "4bceff59001d8a0b8afc6af1cc425997"
server_key_version: "encryptionkey@0"
I20260619 06:50:53.946002 28468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:50:53.948213 28483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:53.949275 28468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:50:53.949422 28468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "37d50280f05e4646ab8bae3d97c2267d"
format_stamp: "Formatted at 2026-06-19 06:50:53 on dist-test-slave-9x3b"
server_key: "5b843e2a7a9521fb159e998b08ebf236"
server_key_iv: "4bceff59001d8a0b8afc6af1cc425997"
server_key_version: "encryptionkey@0"
I20260619 06:50:53.949532 28468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:53.977814 28468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:53.980858 28468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:53.981069 28468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:53.987856 28468 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:44369
I20260619 06:50:53.987884 28535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:44369 every 8 connection(s)
I20260619 06:50:53.988749 28468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:50:53.991386 28536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:53.995311 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 28468
I20260619 06:50:53.995415 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:50:53.995764 14803 external_mini_cluster.cc:1428] Setting key 71ae140050bf0bd13fb4b3a122c1d81c
I20260619 06:50:53.996887 28536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d: Bootstrap starting.
I20260619 06:50:53.999028 28536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:53.999802 28536 log.cc:824] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:54.001507 28536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d: No bootstrap required, opened a new log
Jun 19 06:50:54 dist-test-slave-9x3b krb5kdc[28455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:54.005575 28536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d50280f05e4646ab8bae3d97c2267d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44369 } }
I20260619 06:50:54.005839 28536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:54.005898 28536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d50280f05e4646ab8bae3d97c2267d, State: Initialized, Role: FOLLOWER
I20260619 06:50:54.006385 28536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d50280f05e4646ab8bae3d97c2267d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44369 } }
I20260619 06:50:54.006558 28536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:50:54.006625 28536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:50:54.006727 28536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:54.007786 28536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d50280f05e4646ab8bae3d97c2267d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44369 } }
I20260619 06:50:54.008113 28536 leader_election.cc:302] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37d50280f05e4646ab8bae3d97c2267d; no voters:
I20260619 06:50:54.008419 28536 leader_election.cc:288] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:50:54.008550 28541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:50:54.008802 28541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [term 1 LEADER]: Becoming Leader. State: Replica: 37d50280f05e4646ab8bae3d97c2267d, State: Running, Role: LEADER
I20260619 06:50:54.009131 28541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d50280f05e4646ab8bae3d97c2267d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44369 } }
I20260619 06:50:54.009639 28536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:50:54.010301 28541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d50280f05e4646ab8bae3d97c2267d. Latest consensus state: current_term: 1 leader_uuid: "37d50280f05e4646ab8bae3d97c2267d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d50280f05e4646ab8bae3d97c2267d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44369 } } }
I20260619 06:50:54.010496 28541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:54.010707 28542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37d50280f05e4646ab8bae3d97c2267d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d50280f05e4646ab8bae3d97c2267d" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 44369 } } }
I20260619 06:50:54.010814 28542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:54.010828 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:53.997069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.14.116.254:44369)
0619 06:50:53.997474 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:53.997489 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:53.997522 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:50:53.998135 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:53.998151 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:53.998455 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:53.998781 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:53.999571 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:54.000851 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:54.001712 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:54.002353 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:54.002556 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:54.005235 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:54.005240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:54.005255 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:54.005282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:54.007316 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:54.007714 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:54.007723 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:54.007731 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:54.007866 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:54.008232 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:54.008236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:54.008237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:54.008727 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:54.009463 (+ 736us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:54.009601 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 06:50:54.009871 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
I20260619 06:50:54.011514 28550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:50:54.013967 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:50:54.015013 28550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:50:54.020148 28550 catalog_manager.cc:1347] Generated new cluster ID: 42f55a752f8640deae046b14c0a418a4
I20260619 06:50:54.020210 28550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:50:54.053738 28550 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:50:54.054745 28550 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:50:54.067598 28550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d50280f05e4646ab8bae3d97c2267d: Generated new TSK 0
I20260619 06:50:54.068400 28550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:50:54.120831 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 28468
2026-06-19T06:50:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1464 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:50:54 dist-test-slave-9x3b krb5kdc[28563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:50:54 dist-test-slave-9x3b krb5kdc[28563](info): set up 2 sockets
Jun 19 06:50:54 dist-test-slave-9x3b krb5kdc[28563](info): commencing operation
krb5kdc: starting...
W20260619 06:50:55.151854 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:50:55 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:50:55.263513 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46023
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:55.368083 28576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:55.368348 28576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:55.368414 28576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:55.372355 28576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:55.372427 28576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:55.372462 28576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:55.372501 28576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:55.372534 28576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:55.377502 28576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:55.378432 28576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:55.379357 28576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:55.387465 28582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:55.387465 28584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:55.387638 28576 server_base.cc:1034] running on GCE node
W20260619 06:50:55.387776 28581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:55.388406 28576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:55.389346 28576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:55.390502 28576 hybrid_clock.cc:648] HybridClock initialized: now 1781851855390486 us; error 28 us; skew 500 ppm
Jun 19 06:50:55 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851855, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:55.393028 28576 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:50:55.393886 28576 webserver.cc:460] Webserver started at http://127.14.116.254:35471/ using document root <none> and password file <none>
I20260619 06:50:55.394402 28576 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:55.394459 28576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:55.394611 28576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:50:55.396559 28576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "a89f99fdd4c1470dbaa49ea9a365512e"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5cf004bc386a4c770e7aa3362370927a"
server_key_iv: "5cbb0205efb4ad1fb6f12df34b37ea69"
server_key_version: "encryptionkey@0"
I20260619 06:50:55.397042 28576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "a89f99fdd4c1470dbaa49ea9a365512e"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5cf004bc386a4c770e7aa3362370927a"
server_key_iv: "5cbb0205efb4ad1fb6f12df34b37ea69"
server_key_version: "encryptionkey@0"
I20260619 06:50:55.400516 28576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:50:55.402643 28591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:55.403829 28576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:50:55.403959 28576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "a89f99fdd4c1470dbaa49ea9a365512e"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5cf004bc386a4c770e7aa3362370927a"
server_key_iv: "5cbb0205efb4ad1fb6f12df34b37ea69"
server_key_version: "encryptionkey@0"
I20260619 06:50:55.404053 28576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:55.427863 28576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:55.431021 28576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:55.431221 28576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:55.437875 28576 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:46023
I20260619 06:50:55.438241 28643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:46023 every 8 connection(s)
I20260619 06:50:55.438809 28576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:50:55.441344 28644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:55.446336 28644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Bootstrap starting.
I20260619 06:50:55.448571 28644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:55.448962 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 28576
I20260619 06:50:55.449102 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:50:55.449350 28644 log.cc:824] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:55.449378 14803 external_mini_cluster.cc:1428] Setting key 76da2e961240665d2450891c095ab850
I20260619 06:50:55.451231 28644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: No bootstrap required, opened a new log
I20260619 06:50:55.453998 28644 raft_consensus.cc:348] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:55.454231 28644 raft_consensus.cc:374] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:55.454314 28644 raft_consensus.cc:729] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89f99fdd4c1470dbaa49ea9a365512e, State: Initialized, Role: FOLLOWER
I20260619 06:50:55.454752 28644 consensus_queue.cc:260] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:55.454902 28644 raft_consensus.cc:388] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:50:55.454991 28644 raft_consensus.cc:482] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:50:55.455084 28644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:55.455952 28644 raft_consensus.cc:504] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:55.456267 28644 leader_election.cc:302] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a89f99fdd4c1470dbaa49ea9a365512e; no voters:
I20260619 06:50:55.456569 28644 leader_election.cc:288] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:50:55.456664 28649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:50:55.456843 28649 raft_consensus.cc:686] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 1 LEADER]: Becoming Leader. State: Replica: a89f99fdd4c1470dbaa49ea9a365512e, State: Running, Role: LEADER
I20260619 06:50:55.457122 28649 consensus_queue.cc:237] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:55.457736 28644 sys_catalog.cc:564] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 06:50:55 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:55.458482 28651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a89f99fdd4c1470dbaa49ea9a365512e. Latest consensus state: current_term: 1 leader_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } } }
I20260619 06:50:55.458622 28651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:55.459401 28650 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } } }
I20260619 06:50:55.459937 28650 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: This master's current role is: LEADER
W20260619 06:50:55.463024 28664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:50:55.463101 28664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:50:55.463680 28658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:50:55.464331 28658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:50:55.464387 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:55.450391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.14.116.254:46023)
0619 06:50:55.450778 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:55.450792 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:55.450824 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:50:55.451476 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:55.451492 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:55.451846 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:55.452160 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:55.452900 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.454283 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.455316 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.455848 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.456024 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:55.458922 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:55.458928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:55.458943 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:55.458975 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:55.461319 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:55.461785 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:55.461790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:55.461796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:55.461940 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:55.462239 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:55.462242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:55.462244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:55.462717 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:55.462916 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:55.463200 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 06:50:55.463485 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20260619 06:50:55.469440 28658 catalog_manager.cc:1347] Generated new cluster ID: 01da7e66ac4f42d0806f18639afec59c
I20260619 06:50:55.469502 28658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:50:55.490809 28658 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:50:55.492056 28658 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:50:55.502182 28658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Generated new TSK 0
I20260619 06:50:55.502847 28658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:50:55.534634 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46023
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:55.642004 28670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:55.642232 28670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:55.642290 28670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:55.646240 28670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:55.646296 28670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:55.646350 28670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:50:55.651284 28670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:55.652256 28670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:55.653162 28670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:55.663272 28678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:55.663259 28675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:55.663303 28670 server_base.cc:1034] running on GCE node
W20260619 06:50:55.663266 28676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:55.663865 28670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:55.664458 28670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:55.665612 28670 hybrid_clock.cc:648] HybridClock initialized: now 1781851855665602 us; error 23 us; skew 500 ppm
Jun 19 06:50:55 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851855, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:55.668152 28670 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:50:55.668917 28670 webserver.cc:460] Webserver started at http://127.14.116.193:46251/ using document root <none> and password file <none>
I20260619 06:50:55.669450 28670 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:55.669541 28670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:55.669773 28670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:50:55.671736 28670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "22c8af2d6502469195259a79df948e46"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "a6db6810da1270945c0e8f161f26b17e"
server_key_iv: "cacc05ef4296604e783998d0242e5886"
server_key_version: "encryptionkey@0"
I20260619 06:50:55.672274 28670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "22c8af2d6502469195259a79df948e46"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "a6db6810da1270945c0e8f161f26b17e"
server_key_iv: "cacc05ef4296604e783998d0242e5886"
server_key_version: "encryptionkey@0"
I20260619 06:50:55.675722 28670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:50:55.677981 28685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:55.679098 28670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:50:55.679224 28670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "22c8af2d6502469195259a79df948e46"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "a6db6810da1270945c0e8f161f26b17e"
server_key_iv: "cacc05ef4296604e783998d0242e5886"
server_key_version: "encryptionkey@0"
I20260619 06:50:55.679329 28670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:55.689831 28670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:55.692427 28670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:55.692612 28670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:55.693138 28670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:55.693885 28670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:55.693954 28670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:55.694063 28670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:55.694093 28670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:55.702544 28670 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:38417
I20260619 06:50:55.702564 28798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:38417 every 8 connection(s)
I20260619 06:50:55.703336 28670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:50:55.710399 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 28670
I20260619 06:50:55.710532 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:50:55.710893 14803 external_mini_cluster.cc:1428] Setting key 8cf1423af0385abe7624a53c350c9b54
Jun 19 06:50:55 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851855, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:55.719714 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:55.705022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:58591 (local address 127.14.116.254:46023)
0619 06:50:55.705148 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:55.705152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:55.706037 (+ 885us) server_negotiation.cc:407] Connection header received
0619 06:50:55.708150 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:55.708156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:55.708220 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:55.708337 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:55.710116 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.711046 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.712676 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.713105 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.713215 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:55.716122 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:55.716125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:55.716128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:55.716153 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:55.717622 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:55.718180 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:55.718183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:55.718185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:55.718263 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:55.718613 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:55.718616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:55.718618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:55.718792 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:55.718902 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:55.719387 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 06:50:55.719509 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:50:55.720526 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:55.705362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:58591)
0619 06:50:55.705846 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:55.705911 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:55.707872 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:55.708567 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:55.708575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:55.709156 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:55.709917 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:55.709934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.711198 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.711201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:55.712469 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:55.712481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.713385 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.713389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:55.713607 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:55.714286 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:55.714309 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:55.715965 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:55.717758 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:55.717764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:55.717778 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:55.718088 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:55.718371 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:55.718374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:55.718376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:55.718514 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:55.718901 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:55.718908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:55.719146 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 06:50:55.719324 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 06:50:55.719649 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260619 06:50:55.721792 28799 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46023
I20260619 06:50:55.722039 28799 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:55.722537 28799 heartbeater.cc:511] Master 127.14.116.254:46023 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
I20260619 06:50:55.724069 28608 ts_manager.cc:194] Registered new tserver with Master: 22c8af2d6502469195259a79df948e46 (127.14.116.193:38417)
I20260619 06:50:55.725296 28608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:58591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:50:55.742257 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46023
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:55.868160 28804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:55.868417 28804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:55.868497 28804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:55.872632 28804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:55.872689 28804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:55.872792 28804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:50:55.877991 28804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:55.879046 28804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:55.880119 28804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:55.889870 28810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:55.889871 28809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:55.889871 28812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:55.890182 28804 server_base.cc:1034] running on GCE node
I20260619 06:50:55.890717 28804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:55.891327 28804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:55.892485 28804 hybrid_clock.cc:648] HybridClock initialized: now 1781851855892449 us; error 47 us; skew 500 ppm
Jun 19 06:50:55 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851855, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:55.895277 28804 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:50:55.896124 28804 webserver.cc:460] Webserver started at http://127.14.116.194:39731/ using document root <none> and password file <none>
I20260619 06:50:55.896657 28804 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:55.896744 28804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:55.896967 28804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:50:55.898900 28804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "b437bff093e54a8b8ec74a1576770cb0"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5d02f2ca0fadec036eb91fc567b44993"
server_key_iv: "3ed3dcf7b0fc68fccbf749346c8a25f7"
server_key_version: "encryptionkey@0"
I20260619 06:50:55.899457 28804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "b437bff093e54a8b8ec74a1576770cb0"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5d02f2ca0fadec036eb91fc567b44993"
server_key_iv: "3ed3dcf7b0fc68fccbf749346c8a25f7"
server_key_version: "encryptionkey@0"
I20260619 06:50:55.903067 28804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:50:55.905411 28819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:55.906607 28804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:50:55.906729 28804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "b437bff093e54a8b8ec74a1576770cb0"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5d02f2ca0fadec036eb91fc567b44993"
server_key_iv: "3ed3dcf7b0fc68fccbf749346c8a25f7"
server_key_version: "encryptionkey@0"
I20260619 06:50:55.906832 28804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:55.944074 28804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:55.947361 28804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:55.947576 28804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:55.948179 28804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:55.948990 28804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:55.949059 28804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:55.949126 28804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:55.949174 28804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:55.957832 28804 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:35251
I20260619 06:50:55.957909 28932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:35251 every 8 connection(s)
I20260619 06:50:55.958902 28804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:50:55.960673 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 28804
I20260619 06:50:55.960819 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:50:55.961143 14803 external_mini_cluster.cc:1428] Setting key 7728d8e02587c629449335ef4d9e63b9
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
Jun 19 06:50:55 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851855, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:55.978080 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:55.960979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:43459 (local address 127.14.116.254:46023)
0619 06:50:55.961108 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:55.961111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:55.962112 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 06:50:55.963967 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:55.963973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:55.964028 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:55.964107 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:55.965403 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.966257 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.967485 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.967937 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.968073 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:55.974202 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:55.974205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:55.974208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:55.974242 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:55.975929 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:55.976409 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:55.976415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:55.976420 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:55.976519 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:55.976836 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:55.976842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:55.976847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:55.977111 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:55.977226 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:55.977692 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 06:50:55.977837 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:50:55.978780 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:55.961461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:43459)
0619 06:50:55.961947 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:55.962000 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:55.963742 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:55.964310 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:55.964321 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:55.964795 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:55.965262 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:55.965275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.966386 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.966388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:55.967349 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:55.967365 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:55.968084 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:55.968087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:55.968231 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:55.972373 (+ 4142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:55.972391 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:55.974020 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:55.976038 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:55.976043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:55.976057 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:55.976306 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:55.976610 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:55.976611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:55.976613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:55.976738 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:55.977213 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:55.977218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:55.977451 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 06:50:55.977607 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 06:50:55.977865 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":135,"threads_started":1}
I20260619 06:50:55.980051 28933 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46023
I20260619 06:50:55.980314 28933 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:55.980931 28933 heartbeater.cc:511] Master 127.14.116.254:46023 requested a full tablet report, sending...
I20260619 06:50:55.982105 28608 ts_manager.cc:194] Registered new tserver with Master: b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251)
I20260619 06:50:55.982731 28608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:43459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:50:55.988525 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46023
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:56.096436 28938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:56.096668 28938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:56.096709 28938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:56.100533 28938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:56.100579 28938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:56.100634 28938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:50:56.105533 28938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:56.106462 28938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:56.107435 28938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:56.116804 28946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:56.116849 28944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:56.116981 28938 server_base.cc:1034] running on GCE node
W20260619 06:50:56.116811 28943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:56.117565 28938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:56.118137 28938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:56.119294 28938 hybrid_clock.cc:648] HybridClock initialized: now 1781851856119278 us; error 28 us; skew 500 ppm
Jun 19 06:50:56 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851856, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:56.122052 28938 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:50:56.122864 28938 webserver.cc:460] Webserver started at http://127.14.116.195:43375/ using document root <none> and password file <none>
I20260619 06:50:56.123405 28938 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:56.123512 28938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:56.123766 28938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:50:56.125739 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "3bac9c5cf17a4ddca7aaab6c527435d3"
format_stamp: "Formatted at 2026-06-19 06:50:56 on dist-test-slave-9x3b"
server_key: "6612e51372b03974e7ce5950b2a86534"
server_key_iv: "63eaee314e91c14db82bc8328b9640c3"
server_key_version: "encryptionkey@0"
I20260619 06:50:56.126276 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "3bac9c5cf17a4ddca7aaab6c527435d3"
format_stamp: "Formatted at 2026-06-19 06:50:56 on dist-test-slave-9x3b"
server_key: "6612e51372b03974e7ce5950b2a86534"
server_key_iv: "63eaee314e91c14db82bc8328b9640c3"
server_key_version: "encryptionkey@0"
I20260619 06:50:56.129858 28938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:50:56.132194 28953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:56.133395 28938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:50:56.133517 28938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "3bac9c5cf17a4ddca7aaab6c527435d3"
format_stamp: "Formatted at 2026-06-19 06:50:56 on dist-test-slave-9x3b"
server_key: "6612e51372b03974e7ce5950b2a86534"
server_key_iv: "63eaee314e91c14db82bc8328b9640c3"
server_key_version: "encryptionkey@0"
I20260619 06:50:56.133646 28938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:56.156188 28938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:56.159754 28938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:56.159955 28938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:56.160534 28938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:56.161347 28938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:56.161396 28938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:56.161465 28938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:56.161496 28938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:56.169872 28938 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:38043
I20260619 06:50:56.169890 29066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:38043 every 8 connection(s)
I20260619 06:50:56.170723 28938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:50:56.174333 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 28938
I20260619 06:50:56.174435 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:50:56.174729 14803 external_mini_cluster.cc:1428] Setting key 4c38cf39589a135ecde4737a98824f1e
Jun 19 06:50:56 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851856, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:56.183663 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.172478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:58957 (local address 127.14.116.254:46023)
0619 06:50:56.172615 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.172619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.173354 (+ 735us) server_negotiation.cc:407] Connection header received
0619 06:50:56.174772 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.174775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.174834 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.174933 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:56.176113 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.176639 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.177850 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.178076 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.178185 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.180500 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:56.180503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:56.180506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:56.180537 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:56.181724 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.182203 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.182208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.182210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.182290 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.182621 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.182626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.182628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.182809 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:56.182915 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.183367 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.183480 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:50:56.184521 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.172770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:58957)
0619 06:50:56.173199 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.173239 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:56.174576 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:56.175112 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:56.175119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:56.175457 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:56.175977 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.175991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.176758 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.176761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.177736 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.177746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.178216 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.178221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.178384 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.178891 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:56.178906 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:56.180334 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:56.181835 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:56.181841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:56.181854 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:56.182103 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:56.182378 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:56.182383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:56.182385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:56.182520 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:56.182911 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:56.182917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:56.183149 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 06:50:56.183329 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 06:50:56.183682 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260619 06:50:56.185503 29067 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46023
I20260619 06:50:56.185727 29067 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:56.186213 29067 heartbeater.cc:511] Master 127.14.116.254:46023 requested a full tablet report, sending...
I20260619 06:50:56.187386 28608 ts_manager.cc:194] Registered new tserver with Master: 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043)
I20260619 06:50:56.188078 28608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:58957
I20260619 06:50:56.188512 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:50:56.197489 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.190395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.14.116.254:46023)
0619 06:50:56.190529 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.190536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.190755 (+ 219us) server_negotiation.cc:407] Connection header received
0619 06:50:56.190836 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.190838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.190871 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.190939 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:56.191554 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.192089 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.192833 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.193032 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.193102 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.194267 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:56.194270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:56.194272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:56.194291 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:56.195854 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.196284 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.196287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.196289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.196368 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.196639 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.196642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.196643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.196846 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:56.196990 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.197211 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.197319 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:50:56.199929 28608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:50:56.202768 28608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:50:56.217060 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.210908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:35251 (local address 127.0.0.1:54814)
0619 06:50:56.211232 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.211274 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:56.211389 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:56.212738 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:56.212744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:56.212767 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:56.212959 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.212968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.214418 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.214421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.215497 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.215503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.216488 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.216491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.216588 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.216663 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 06:50:56.216743 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:50:56.216883 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":85,"thread_start_us":62,"threads_started":1}
I20260619 06:50:56.217068 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.211631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.14.116.194:35251)
0619 06:50:56.212287 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.212302 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.212320 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:50:56.212391 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.212396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.212547 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.212715 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:56.213094 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.214279 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.215629 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.216345 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.216673 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.216798 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.216848 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.216937 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":125,"threads_started":1}
I20260619 06:50:56.217218 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.211536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.14.116.193:38417)
0619 06:50:56.211850 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.211866 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.212157 (+ 291us) server_negotiation.cc:407] Connection header received
0619 06:50:56.212241 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.212246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.212468 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.212618 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:56.213169 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.213884 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.215442 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.216310 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.216447 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.216585 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.216967 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.217060 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":122,"threads_started":1}
I20260619 06:50:56.218240 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.211585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:38043 (local address 127.0.0.1:56798)
0619 06:50:56.211987 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.212000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:56.212065 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:56.212878 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:56.212880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:56.212892 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:56.213015 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.213022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.215589 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.215591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.216347 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.216363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.217819 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.217823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.217916 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.217938 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:50:56.218015 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:50:56.218084 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":27,"thread_start_us":149,"threads_started":1}
I20260619 06:50:56.218622 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.211025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:38417 (local address 127.0.0.1:40122)
0619 06:50:56.211946 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.211960 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:56.212060 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:56.212805 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:56.212808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:56.212828 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:56.213017 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.213022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.214018 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.214024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.215326 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.215336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.216443 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.216449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.216559 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.216646 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 06:50:56.216750 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:50:56.216890 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":67,"thread_start_us":132,"threads_started":1}
I20260619 06:50:56.218848 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.212147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.14.116.195:38043)
0619 06:50:56.212435 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.212448 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.212468 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:50:56.212542 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.212547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.212755 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.212910 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:56.213113 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.214279 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.216516 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.217697 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.217794 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.217877 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.218169 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.218264 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
I20260619 06:50:56.220212 28867 tablet_service.cc:1467] Processing CreateTablet for tablet 9b918f43adb24e238ad2bedb12e050da (DEFAULT_TABLE table=test-table [id=8742adf1bed34d6da59bcc7341ed8379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:56.221228 28867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b918f43adb24e238ad2bedb12e050da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:56.221231 29001 tablet_service.cc:1467] Processing CreateTablet for tablet 9b918f43adb24e238ad2bedb12e050da (DEFAULT_TABLE table=test-table [id=8742adf1bed34d6da59bcc7341ed8379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:56.221230 28733 tablet_service.cc:1467] Processing CreateTablet for tablet 9b918f43adb24e238ad2bedb12e050da (DEFAULT_TABLE table=test-table [id=8742adf1bed34d6da59bcc7341ed8379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:56.222198 28733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b918f43adb24e238ad2bedb12e050da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:56.222204 29001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b918f43adb24e238ad2bedb12e050da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:56.227397 29084 tablet_bootstrap.cc:492] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Bootstrap starting.
I20260619 06:50:56.228905 29085 tablet_bootstrap.cc:492] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Bootstrap starting.
I20260619 06:50:56.229759 29086 tablet_bootstrap.cc:492] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: Bootstrap starting.
I20260619 06:50:56.230170 29084 tablet_bootstrap.cc:654] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:56.231034 29084 log.cc:824] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:56.231072 29085 tablet_bootstrap.cc:654] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:56.231482 29086 tablet_bootstrap.cc:654] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:56.231865 29085 log.cc:824] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:56.232713 29086 log.cc:824] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:56.233594 29084 tablet_bootstrap.cc:492] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: No bootstrap required, opened a new log
I20260619 06:50:56.233594 29085 tablet_bootstrap.cc:492] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: No bootstrap required, opened a new log
I20260619 06:50:56.233779 29085 ts_tablet_manager.cc:1397] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:50:56.233779 29084 ts_tablet_manager.cc:1397] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 06:50:56.234340 29086 tablet_bootstrap.cc:492] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: No bootstrap required, opened a new log
I20260619 06:50:56.234520 29086 ts_tablet_manager.cc:1397] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:50:56.237416 29086 raft_consensus.cc:348] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.237650 29086 raft_consensus.cc:374] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:56.237703 29086 raft_consensus.cc:729] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bac9c5cf17a4ddca7aaab6c527435d3, State: Initialized, Role: FOLLOWER
I20260619 06:50:56.237725 29084 raft_consensus.cc:348] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.237963 29084 raft_consensus.cc:374] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:56.238031 29084 raft_consensus.cc:729] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b437bff093e54a8b8ec74a1576770cb0, State: Initialized, Role: FOLLOWER
I20260619 06:50:56.238009 29085 raft_consensus.cc:348] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.238227 29085 raft_consensus.cc:374] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:56.238161 29086 consensus_queue.cc:260] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.238284 29085 raft_consensus.cc:729] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c8af2d6502469195259a79df948e46, State: Initialized, Role: FOLLOWER
I20260619 06:50:56.238480 29084 consensus_queue.cc:260] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.238633 29085 consensus_queue.cc:260] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.239236 29086 ts_tablet_manager.cc:1428] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:50:56.239459 29085 ts_tablet_manager.cc:1428] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260619 06:50:56.239565 29067 heartbeater.cc:503] Master 127.14.116.254:46023 was elected leader, sending a full tablet report...
I20260619 06:50:56.239566 28933 heartbeater.cc:503] Master 127.14.116.254:46023 was elected leader, sending a full tablet report...
I20260619 06:50:56.239784 28799 heartbeater.cc:503] Master 127.14.116.254:46023 was elected leader, sending a full tablet report...
I20260619 06:50:56.240494 29084 ts_tablet_manager.cc:1428] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 06:50:56.252676 29091 raft_consensus.cc:482] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:50:56.252851 29091 raft_consensus.cc:504] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.253947 29091 leader_election.cc:288] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22c8af2d6502469195259a79df948e46 (127.14.116.193:38417), 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043)
I20260619 06:50:56.258886 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.254258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:59027 (local address 127.14.116.193:38417)
0619 06:50:56.254392 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.254396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.254718 (+ 322us) server_negotiation.cc:407] Connection header received
0619 06:50:56.254790 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.254792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.254836 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.254911 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:56.255164 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.255895 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.257814 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.258361 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.258458 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.258496 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.258695 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.258767 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:50:56.258908 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.254161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:38417 (local address 127.14.116.194:59027)
0619 06:50:56.254633 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.254643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:56.254719 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:56.254939 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:56.254942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:56.254954 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:56.255062 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.255075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.256004 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.256007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.257713 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.257723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.258457 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.258460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.258550 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.258589 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:50:56.258677 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:50:56.258749 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":245,"thread_start_us":128,"threads_started":1}
I20260619 06:50:56.259775 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.254157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:38043 (local address 127.14.116.194:45069)
0619 06:50:56.254273 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.254288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:56.254376 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:56.254804 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:56.254806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:56.254815 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:56.254956 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.254962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.256004 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.256007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.257713 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.257723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.259339 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.259342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.259484 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.259501 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 06:50:56.259571 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:50:56.259673 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":246}
I20260619 06:50:56.259704 28753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b918f43adb24e238ad2bedb12e050da" candidate_uuid: "b437bff093e54a8b8ec74a1576770cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c8af2d6502469195259a79df948e46" is_pre_election: true
I20260619 06:50:56.259845 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.254480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:45069 (local address 127.14.116.195:38043)
0619 06:50:56.254599 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.254602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.254614 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:50:56.254653 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.254655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.254713 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.254792 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:56.255066 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.255889 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.258567 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.259232 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.259352 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.259404 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.259639 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.259729 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:50:56.260025 28753 raft_consensus.cc:2454] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b437bff093e54a8b8ec74a1576770cb0 in term 0.
I20260619 06:50:56.260447 29021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b918f43adb24e238ad2bedb12e050da" candidate_uuid: "b437bff093e54a8b8ec74a1576770cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" is_pre_election: true
I20260619 06:50:56.260506 28820 leader_election.cc:302] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22c8af2d6502469195259a79df948e46, b437bff093e54a8b8ec74a1576770cb0; no voters:
I20260619 06:50:56.260715 29021 raft_consensus.cc:2454] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b437bff093e54a8b8ec74a1576770cb0 in term 0.
I20260619 06:50:56.260771 29091 raft_consensus.cc:2781] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:50:56.260885 29091 raft_consensus.cc:482] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:50:56.260938 29091 raft_consensus.cc:3037] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:56.261929 29091 raft_consensus.cc:504] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.262216 29091 leader_election.cc:288] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [CANDIDATE]: Term 1 election: Requested vote from peers 22c8af2d6502469195259a79df948e46 (127.14.116.193:38417), 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043)
I20260619 06:50:56.262508 29021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b918f43adb24e238ad2bedb12e050da" candidate_uuid: "b437bff093e54a8b8ec74a1576770cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3"
I20260619 06:50:56.262637 29021 raft_consensus.cc:3037] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:56.262652 28753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b918f43adb24e238ad2bedb12e050da" candidate_uuid: "b437bff093e54a8b8ec74a1576770cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c8af2d6502469195259a79df948e46"
I20260619 06:50:56.262745 28753 raft_consensus.cc:3037] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:56.263914 29021 raft_consensus.cc:2454] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b437bff093e54a8b8ec74a1576770cb0 in term 1.
I20260619 06:50:56.264055 28753 raft_consensus.cc:2454] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b437bff093e54a8b8ec74a1576770cb0 in term 1.
I20260619 06:50:56.264254 28821 leader_election.cc:302] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bac9c5cf17a4ddca7aaab6c527435d3, b437bff093e54a8b8ec74a1576770cb0; no voters:
I20260619 06:50:56.264438 29091 raft_consensus.cc:2781] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:50:56.264616 29091 raft_consensus.cc:686] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 1 LEADER]: Becoming Leader. State: Replica: b437bff093e54a8b8ec74a1576770cb0, State: Running, Role: LEADER
I20260619 06:50:56.264910 29091 consensus_queue.cc:237] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.267890 28607 catalog_manager.cc:5495] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194). New cstate: current_term: 1 leader_uuid: "b437bff093e54a8b8ec74a1576770cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } health_report { overall_health: UNKNOWN } } }
I20260619 06:50:56.277529 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.274517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.14.116.194:35251)
0619 06:50:56.274628 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.274638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.274685 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:50:56.274804 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.274806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.274853 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.274936 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:50:56.275118 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.275588 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.276399 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.276598 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.276671 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.276898 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:50:56.277261 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:50:56.277346 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.277397 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.277439 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:50:56.282778 28753 raft_consensus.cc:1261] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Refusing update from remote peer b437bff093e54a8b8ec74a1576770cb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:56.282821 29021 raft_consensus.cc:1261] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 1 FOLLOWER]: Refusing update from remote peer b437bff093e54a8b8ec74a1576770cb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:56.283447 29091 consensus_queue.cc:1035] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:56.283682 29094 consensus_queue.cc:1035] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:56.293838 29099 mvcc.cc:205] Tried to move back new op lower bound from 7298465203327533056 to 7298465203263385600. Current Snapshot: MvccSnapshot[applied={T|T < 7298465203327533056 or (T in {7298465203327533056})}]
I20260619 06:50:56.300135 28608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:50:56.300277 28608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:50:56.302512 28608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Sending DeleteTablet for 3 replicas of tablet 9b918f43adb24e238ad2bedb12e050da
I20260619 06:50:56.303212 28733 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b918f43adb24e238ad2bedb12e050da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:56 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:40122
I20260619 06:50:56.303247 29001 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b918f43adb24e238ad2bedb12e050da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:56 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:56798
I20260619 06:50:56.303308 28867 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b918f43adb24e238ad2bedb12e050da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:56 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:54814
I20260619 06:50:56.303721 29108 tablet_replica.cc:330] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: stopping tablet replica
I20260619 06:50:56.303723 29107 tablet_replica.cc:330] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: stopping tablet replica
I20260619 06:50:56.304030 29107 raft_consensus.cc:2229] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:50:56.304030 29108 raft_consensus.cc:2229] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:50:56.304132 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 28670
I20260619 06:50:56.305647 29107 raft_consensus.cc:2258] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:56.306020 29109 tablet_replica.cc:330] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: stopping tablet replica
I20260619 06:50:56.306269 29109 raft_consensus.cc:2229] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:50:56.306610 29109 raft_consensus.cc:2258] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:56.307312 29107 ts_tablet_manager.cc:1905] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:56.308900 29109 ts_tablet_manager.cc:1905] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:56.310741 29107 ts_tablet_manager.cc:1918] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:56.310866 29107 log.cc:1196] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/9b918f43adb24e238ad2bedb12e050da
I20260619 06:50:56.311220 29107 ts_tablet_manager.cc:1939] T 9b918f43adb24e238ad2bedb12e050da P 3bac9c5cf17a4ddca7aaab6c527435d3: Deleting consensus metadata
W20260619 06:50:56.311273 28592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:38417 (error 108)
I20260619 06:50:56.311801 29109 ts_tablet_manager.cc:1918] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:56.311964 29109 log.cc:1196] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/9b918f43adb24e238ad2bedb12e050da
I20260619 06:50:56.312466 28593 catalog_manager.cc:4841] TS 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043): tablet 9b918f43adb24e238ad2bedb12e050da (table test-table [id=8742adf1bed34d6da59bcc7341ed8379]) successfully deleted
I20260619 06:50:56.312703 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.312385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:38417 (local address 127.0.0.1:40126)
0619 06:50:56.312546 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.312610 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 06:50:56.312767 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 28804
W20260619 06:50:56.313848 28592 catalog_manager.cc:4584] TS 22c8af2d6502469195259a79df948e46 (127.14.116.193:38417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b918f43adb24e238ad2bedb12e050da: Network error: Client connection negotiation failed: client connection to 127.14.116.193:38417: connect: Connection refused (error 111)
I20260619 06:50:56.317830 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 28938
I20260619 06:50:56.318701 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.318058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:35251 (local address 127.0.0.1:54828)
0619 06:50:56.318542 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.318603 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.194:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
I20260619 06:50:56.322973 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 28576
I20260619 06:50:56.328073 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46023
--webserver_interface=127.14.116.254
--webserver_port=35471
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:56.433471 29111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:56.433761 29111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:56.433835 29111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:56.438090 29111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:56.438169 29111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:56.438205 29111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:56.438241 29111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:56.438274 29111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:56.443363 29111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=35471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:56.444344 29111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:56.445295 29111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:56.457608 29120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:56.457810 29111 server_base.cc:1034] running on GCE node
W20260619 06:50:56.457612 29118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:56.457608 29117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:56.458513 29111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:56.459518 29111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:56.460672 29111 hybrid_clock.cc:648] HybridClock initialized: now 1781851856460659 us; error 26 us; skew 500 ppm
Jun 19 06:50:56 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851856, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:56.463325 29111 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:50:56.464195 29111 webserver.cc:460] Webserver started at http://127.14.116.254:35471/ using document root <none> and password file <none>
I20260619 06:50:56.464738 29111 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:56.464836 29111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:56.468417 29111 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:50:56.470340 29127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:56.471485 29111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:50:56.471673 29111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "a89f99fdd4c1470dbaa49ea9a365512e"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5cf004bc386a4c770e7aa3362370927a"
server_key_iv: "5cbb0205efb4ad1fb6f12df34b37ea69"
server_key_version: "encryptionkey@0"
I20260619 06:50:56.472230 29111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:56.496833 29111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:56.499950 29111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:56.500150 29111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:56.506906 29111 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:46023
I20260619 06:50:56.507185 29179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:46023 every 8 connection(s)
I20260619 06:50:56.507974 29111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:50:56.512166 29180 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:50:56.513584 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 29111
I20260619 06:50:56.513769 29180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Bootstrap starting.
I20260619 06:50:56.514046 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:38417
--local_ip_for_outbound_sockets=127.14.116.193
--tserver_master_addrs=127.14.116.254:46023
--webserver_port=46251
--webserver_interface=127.14.116.193
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:50:56.536583 29180 log.cc:824] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:56.547127 29180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:50:56.547523 29180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Bootstrap complete.
I20260619 06:50:56.552778 29180 raft_consensus.cc:348] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:56.553267 29180 raft_consensus.cc:729] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89f99fdd4c1470dbaa49ea9a365512e, State: Initialized, Role: FOLLOWER
I20260619 06:50:56.553793 29180 consensus_queue.cc:260] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:56.553939 29180 raft_consensus.cc:388] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:50:56.554008 29180 raft_consensus.cc:482] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:50:56.554098 29180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:50:56.556567 29180 raft_consensus.cc:504] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:56.556931 29180 leader_election.cc:302] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a89f99fdd4c1470dbaa49ea9a365512e; no voters:
I20260619 06:50:56.557248 29180 leader_election.cc:288] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:50:56.557372 29184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:50:56.557634 29184 raft_consensus.cc:686] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 2 LEADER]: Becoming Leader. State: Replica: a89f99fdd4c1470dbaa49ea9a365512e, State: Running, Role: LEADER
I20260619 06:50:56.557922 29184 consensus_queue.cc:237] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:56.558430 29180 sys_catalog.cc:564] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:50:56.559425 29185 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } } }
I20260619 06:50:56.559579 29185 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:56.559703 29186 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a89f99fdd4c1470dbaa49ea9a365512e. Latest consensus state: current_term: 2 leader_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } } }
I20260619 06:50:56.559798 29186 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:56.560242 29192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:50:56.565033 29192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:50:56.565817 29192 catalog_manager.cc:1259] Loaded cluster ID: 01da7e66ac4f42d0806f18639afec59c
I20260619 06:50:56.565887 29192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:50:56.566936 29192 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 06:50:56.569854 29192 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 06:50:56.576709 29192 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:50:56.577291 29192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Loaded TSK: 0
I20260619 06:50:56.578179 29192 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:50:56.644312 29182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:56.644572 29182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:56.644641 29182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:56.648576 29182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:56.648651 29182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:56.648748 29182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:50:56.653843 29182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:38417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=46251
--tserver_master_addrs=127.14.116.254:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:56.654798 29182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:56.655776 29182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:56.667027 29208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:56.667028 29207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:56.667158 29182 server_base.cc:1034] running on GCE node
W20260619 06:50:56.667251 29210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:56.667877 29182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:56.668489 29182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:56.669672 29182 hybrid_clock.cc:648] HybridClock initialized: now 1781851856669646 us; error 44 us; skew 500 ppm
Jun 19 06:50:56 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851856, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:56.672425 29182 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:50:56.673344 29182 webserver.cc:460] Webserver started at http://127.14.116.193:46251/ using document root <none> and password file <none>
I20260619 06:50:56.673874 29182 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:56.673976 29182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:56.677896 29182 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:50:56.680035 29217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:56.681300 29182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:50:56.681425 29182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "22c8af2d6502469195259a79df948e46"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "a6db6810da1270945c0e8f161f26b17e"
server_key_iv: "cacc05ef4296604e783998d0242e5886"
server_key_version: "encryptionkey@0"
I20260619 06:50:56.681933 29182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:56.712839 29182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:56.715590 29182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:56.715813 29182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:56.716396 29182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:56.717396 29224 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 06:50:56.721294 29182 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:50:56.721350 29182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 06:50:56.721423 29182 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:50:56.722952 29182 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:50:56.722994 29182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 06:50:56.723248 29224 tablet_bootstrap.cc:492] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Bootstrap starting.
I20260619 06:50:56.733331 29182 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:38417
I20260619 06:50:56.733430 29331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:38417 every 8 connection(s)
I20260619 06:50:56.734261 29182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:50:56.741024 29224 log.cc:824] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:56.741096 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 29182
I20260619 06:50:56.741737 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:35251
--local_ip_for_outbound_sockets=127.14.116.194
--tserver_master_addrs=127.14.116.254:46023
--webserver_port=39731
--webserver_interface=127.14.116.194
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:50:56 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851856, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:56.748704 29224 tablet_bootstrap.cc:492] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:50:56.749073 29224 tablet_bootstrap.cc:492] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Bootstrap complete.
I20260619 06:50:56.750366 29224 ts_tablet_manager.cc:1397] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.004s
I20260619 06:50:56.752938 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.736132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:54335)
0619 06:50:56.736745 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.736794 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:56.737563 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:56.738770 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:56.738776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:56.739102 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:56.739523 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.739536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.741308 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.741312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.742856 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.742871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.743926 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.743931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.744151 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.744821 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:56.744840 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:56.746736 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:56.749508 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:56.749517 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:56.749531 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:56.749900 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:56.750328 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:56.750331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:56.750333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:56.750494 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:56.751245 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:56.751251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:56.751520 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 06:50:56.751728 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 06:50:56.752042 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":144,"threads_started":1}
I20260619 06:50:56.753049 29224 raft_consensus.cc:348] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.753551 29224 raft_consensus.cc:729] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c8af2d6502469195259a79df948e46, State: Initialized, Role: FOLLOWER
I20260619 06:50:56.754029 29224 consensus_queue.cc:260] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } } peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } }
I20260619 06:50:56.754424 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.736244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:54335 (local address 127.14.116.254:46023)
0619 06:50:56.736615 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.736634 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.736926 (+ 292us) server_negotiation.cc:407] Connection header received
0619 06:50:56.737966 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.737981 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.738355 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.738763 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:56.739720 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.741129 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.743002 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.743761 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.743942 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.746973 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:56.746983 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:56.747002 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:56.747041 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:56.749384 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.750025 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.750034 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.750043 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.750203 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.750601 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.750606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.750609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.751103 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:56.751237 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.753339 (+ 2102us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.753622 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
I20260619 06:50:56.754917 29224 ts_tablet_manager.cc:1428] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260619 06:50:56.756343 29332 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46023
I20260619 06:50:56.756557 29332 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:56.756979 29332 heartbeater.cc:511] Master 127.14.116.254:46023 requested a full tablet report, sending...
I20260619 06:50:56.759147 29144 ts_manager.cc:194] Registered new tserver with Master: 22c8af2d6502469195259a79df948e46 (127.14.116.193:38417)
I20260619 06:50:56.762601 29144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:54335
I20260619 06:50:56.765005 29332 heartbeater.cc:503] Master 127.14.116.254:46023 was elected leader, sending a full tablet report...
Jun 19 06:50:56 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851856, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:50:56.773100 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.761927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:38417 (local address 127.0.0.1:40134)
0619 06:50:56.762950 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.762989 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:56.763113 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:56.763728 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:56.763734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:56.764134 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:56.764251 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.764259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.765569 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.765573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.766479 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.766489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.767183 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.767186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.767274 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.767895 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:56.767907 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:56.769522 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:56.771523 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:56.771530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:56.771534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:56.771808 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:56.772098 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:56.772100 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:56.772102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:56.772173 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:56.772666 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:56.772670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:56.772756 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 06:50:56.772852 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:50:56.772987 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":95,"threads_started":1}
I20260619 06:50:56.773268 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.762520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.14.116.193:38417)
0619 06:50:56.762892 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.762903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.763077 (+ 174us) server_negotiation.cc:407] Connection header received
0619 06:50:56.763259 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.763268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.763466 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.763595 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:56.764383 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.765467 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.766598 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.767076 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.767182 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.769660 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:56.769669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:56.769680 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:56.769705 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:56.771418 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.771911 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.771917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.771919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.772009 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.772275 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.772278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.772280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.772561 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:56.772686 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.772968 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.773122 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":104,"threads_started":1}
I20260619 06:50:56.774103 29266 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b918f43adb24e238ad2bedb12e050da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:56 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:40134
I20260619 06:50:56.775044 29341 tablet_replica.cc:330] stopping tablet replica
I20260619 06:50:56.775333 29341 raft_consensus.cc:2229] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:50:56.775583 29341 raft_consensus.cc:2258] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:56.776569 29341 ts_tablet_manager.cc:1905] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:56.779803 29341 ts_tablet_manager.cc:1918] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:56.779884 29341 log.cc:1196] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/9b918f43adb24e238ad2bedb12e050da
I20260619 06:50:56.780186 29341 ts_tablet_manager.cc:1939] T 9b918f43adb24e238ad2bedb12e050da P 22c8af2d6502469195259a79df948e46: Deleting consensus metadata
I20260619 06:50:56.781641 29128 catalog_manager.cc:4841] TS 22c8af2d6502469195259a79df948e46 (127.14.116.193:38417): tablet 9b918f43adb24e238ad2bedb12e050da (table test-table [id=8742adf1bed34d6da59bcc7341ed8379]) successfully deleted
W20260619 06:50:56.856329 29337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:56.856590 29337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:56.856660 29337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:56.860886 29337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:56.860961 29337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:56.861044 29337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:50:56.866160 29337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:35251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=39731
--tserver_master_addrs=127.14.116.254:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:56.867161 29337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:56.868216 29337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:56.877408 29347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:56.877446 29349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:56.877550 29337 server_base.cc:1034] running on GCE node
W20260619 06:50:56.877409 29346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:56.878036 29337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:56.878629 29337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:56.879783 29337 hybrid_clock.cc:648] HybridClock initialized: now 1781851856879772 us; error 24 us; skew 500 ppm
Jun 19 06:50:56 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851856, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:56.882290 29337 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:50:56.883064 29337 webserver.cc:460] Webserver started at http://127.14.116.194:39731/ using document root <none> and password file <none>
I20260619 06:50:56.883576 29337 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:56.883713 29337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:56.887323 29337 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:50:56.889377 29356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:56.890483 29337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 06:50:56.890604 29337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "b437bff093e54a8b8ec74a1576770cb0"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5d02f2ca0fadec036eb91fc567b44993"
server_key_iv: "3ed3dcf7b0fc68fccbf749346c8a25f7"
server_key_version: "encryptionkey@0"
I20260619 06:50:56.891111 29337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:56.911710 29337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:56.914371 29337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:56.914556 29337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:56.915102 29337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:56.916102 29363 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 06:50:56.919580 29363 ts_tablet_manager.cc:1852] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 06:50:56.920022 29363 ts_tablet_manager.cc:1905] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:56.923477 29363 ts_tablet_manager.cc:1918] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:56.923663 29363 ts_tablet_manager.cc:1939] T 9b918f43adb24e238ad2bedb12e050da P b437bff093e54a8b8ec74a1576770cb0: Deleting consensus metadata
I20260619 06:50:56.923977 29337 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 06:50:56.924047 29337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260619 06:50:56.924132 29337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:56.924178 29337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:56.932476 29337 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:35251
I20260619 06:50:56.932510 29470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:35251 every 8 connection(s)
I20260619 06:50:56.933245 29337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:50:56.938458 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 29337
I20260619 06:50:56.938928 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:38043
--local_ip_for_outbound_sockets=127.14.116.195
--tserver_master_addrs=127.14.116.254:46023
--webserver_port=43375
--webserver_interface=127.14.116.195
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:50:56 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851856, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:56.946626 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.934915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:47011 (local address 127.14.116.254:46023)
0619 06:50:56.935058 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:56.935063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:56.935934 (+ 871us) server_negotiation.cc:407] Connection header received
0619 06:50:56.936743 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:56.936747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:56.936792 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:56.936865 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:56.937930 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.938437 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.940147 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.940530 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.940642 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.943239 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:56.943243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:56.943245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:56.943267 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:56.944703 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.945194 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.945198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.945200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.945280 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:56.945611 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:56.945616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:56.945618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:56.945789 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:56.945895 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:56.946332 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 06:50:56.946457 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:50:56.947345 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:56.935313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:47011)
0619 06:50:56.935784 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:56.935825 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:56.936566 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:56.937020 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:56.937027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:56.937343 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:56.937804 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.937816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.938563 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.938568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.940030 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:56.940044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:56.940685 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:56.940688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:56.940824 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:56.941422 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:56.941438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:56.943076 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:56.944816 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:56.944821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:56.944834 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:56.945082 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:56.945379 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:56.945382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:56.945384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:56.945518 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:56.945890 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:56.945896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:56.946120 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 06:50:56.946293 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 06:50:56.946590 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":115,"threads_started":1}
I20260619 06:50:56.948508 29471 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46023
I20260619 06:50:56.948736 29471 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:56.949188 29471 heartbeater.cc:511] Master 127.14.116.254:46023 requested a full tablet report, sending...
I20260619 06:50:56.950168 29144 ts_manager.cc:194] Registered new tserver with Master: b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251)
I20260619 06:50:56.950898 29144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:47011
W20260619 06:50:57.048424 29474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:57.048646 29474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:57.048686 29474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:57.052486 29474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:57.052539 29474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:57.052593 29474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:50:57.057482 29474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:38043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=43375
--tserver_master_addrs=127.14.116.254:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:57.058322 29474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:57.059334 29474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:57.069931 29480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:57.069957 29482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:57.070148 29479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:57.070182 29474 server_base.cc:1034] running on GCE node
I20260619 06:50:57.070760 29474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:57.071482 29474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:57.072690 29474 hybrid_clock.cc:648] HybridClock initialized: now 1781851857072667 us; error 37 us; skew 500 ppm
Jun 19 06:50:57 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851857, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:57.075579 29474 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:50:57.076483 29474 webserver.cc:460] Webserver started at http://127.14.116.195:43375/ using document root <none> and password file <none>
I20260619 06:50:57.076967 29474 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:57.077033 29474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:57.080952 29474 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:50:57.082793 29489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:57.084053 29474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:50:57.084143 29474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "3bac9c5cf17a4ddca7aaab6c527435d3"
format_stamp: "Formatted at 2026-06-19 06:50:56 on dist-test-slave-9x3b"
server_key: "6612e51372b03974e7ce5950b2a86534"
server_key_iv: "63eaee314e91c14db82bc8328b9640c3"
server_key_version: "encryptionkey@0"
I20260619 06:50:57.084633 29474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:57.103734 29474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:57.106318 29474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:57.106472 29474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:57.106956 29474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:57.107703 29474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:57.107748 29474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:57.107785 29474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:57.107805 29474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:57.116431 29474 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:38043
I20260619 06:50:57.116464 29602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:38043 every 8 connection(s)
I20260619 06:50:57.117370 29474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:50:57.125615 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 29474
Jun 19 06:50:57 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851857, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:50:57.130604 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.119019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:46219 (local address 127.14.116.254:46023)
0619 06:50:57.119166 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:57.119171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:57.120036 (+ 865us) server_negotiation.cc:407] Connection header received
0619 06:50:57.121271 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:57.121274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:57.121324 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:57.121396 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:57.122480 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.122990 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.124126 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.124345 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.124430 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.127005 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:57.127009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:57.127011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:57.127039 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:57.128557 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:57.129082 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:57.129085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:57.129086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:57.129186 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:57.129507 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:57.129514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:57.129517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:57.129725 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:57.129799 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:57.130391 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 06:50:57.130486 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":1792}
I20260619 06:50:57.131480 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.119415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:46219)
0619 06:50:57.119894 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.119934 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:57.121091 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:57.121554 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:57.121560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:57.121895 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:57.122341 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.122354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.123089 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.123092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.124022 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.124031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.124470 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.124472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.124602 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.125104 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:57.125132 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:57.126819 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:57.128658 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:57.128664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:57.128676 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:57.128992 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:57.129272 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:57.129274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:57.129275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:57.129402 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:57.129828 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:57.129835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:57.130123 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 06:50:57.130347 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 06:50:57.130685 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":149,"threads_started":1}
I20260619 06:50:57.132588 29603 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46023
I20260619 06:50:57.132817 29603 heartbeater.cc:464] Registering TS with master...
I20260619 06:50:57.133380 29603 heartbeater.cc:511] Master 127.14.116.254:46023 requested a full tablet report, sending...
I20260619 06:50:57.134465 29144 ts_manager.cc:194] Registered new tserver with Master: 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043)
I20260619 06:50:57.134984 29144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:46219
I20260619 06:50:57.135334 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.126296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.14.116.254:46023)
0619 06:50:57.126572 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:57.126575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:57.126668 (+ 93us) server_negotiation.cc:407] Connection header received
0619 06:50:57.126789 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:57.126791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:57.126825 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:57.126899 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:57.127587 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.128126 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.129135 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.129432 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.129567 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.130956 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:57.130959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:57.130961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:57.130985 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:57.132795 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:57.133226 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:57.133229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:57.133230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:57.133297 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:57.133566 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:57.133568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:57.133569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:57.133714 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:57.133846 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:57.134069 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 06:50:57.134184 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"spinlock_wait_cycles":1920,"thread_start_us":95,"threads_started":1}
I20260619 06:50:57.137004 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:50:57.140621 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.137542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.14.116.254:46023)
0619 06:50:57.137656 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:57.137659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:57.137783 (+ 124us) server_negotiation.cc:407] Connection header received
0619 06:50:57.137903 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:57.137905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:57.137979 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:57.138049 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:50:57.138226 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.138691 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.139447 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.139703 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.139773 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.140009 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:50:57.140361 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:50:57.140447 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:57.140500 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 06:50:57.140543 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:50:57.141220 29144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:50:57.144274 29144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:50:57.151273 29266 tablet_service.cc:1467] Processing CreateTablet for tablet 57b89b853b394680841eefdfa39077c9 (DEFAULT_TABLE table=test-table [id=118d770da198479b81b3c8285a2565f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:57.151947 29266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b89b853b394680841eefdfa39077c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:57.154685 29224 tablet_bootstrap.cc:492] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Bootstrap starting.
I20260619 06:50:57.155308 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.150185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:38043 (local address 127.0.0.1:56802)
0619 06:50:57.150478 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.150493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:57.150563 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:57.151545 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:57.151548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:57.151564 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:57.151695 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.151702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.153110 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.153113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.153992 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.154000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.154919 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.154924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.155020 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.155063 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 06:50:57.155117 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 06:50:57.155185 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":28}
I20260619 06:50:57.155560 29224 tablet_bootstrap.cc:654] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:57.155932 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.150475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.14.116.195:38043)
0619 06:50:57.151067 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:57.151078 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:57.151099 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:50:57.151167 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:57.151172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:57.151382 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:57.151835 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:57.151966 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.152976 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.154109 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.154783 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.155506 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.155672 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:57.155720 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 06:50:57.155801 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":135,"threads_started":1}
I20260619 06:50:57.156904 29224 tablet_bootstrap.cc:492] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: No bootstrap required, opened a new log
I20260619 06:50:57.156989 29224 ts_tablet_manager.cc:1397] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:50:57.157160 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.150535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:35251 (local address 127.0.0.1:54832)
0619 06:50:57.150749 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.150761 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:57.150835 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:57.151680 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:57.151683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:57.151701 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:57.151794 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.151800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.153246 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.153249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.154448 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.154459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.156718 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.156722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.156816 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.156838 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:50:57.156934 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:50:57.157006 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260619 06:50:57.157188 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.150973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.14.116.194:35251)
0619 06:50:57.151241 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:57.151251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:57.151272 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:50:57.151340 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:57.151344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:57.151521 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:57.151711 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:57.151885 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.153115 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.154580 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.156593 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.156721 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.156829 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:57.156963 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 06:50:57.157042 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
I20260619 06:50:57.157472 29224 raft_consensus.cc:348] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.157579 29224 raft_consensus.cc:374] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:57.157620 29224 raft_consensus.cc:729] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c8af2d6502469195259a79df948e46, State: Initialized, Role: FOLLOWER
I20260619 06:50:57.157745 29224 consensus_queue.cc:260] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.158478 29224 ts_tablet_manager.cc:1428] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:50:57.158660 29405 tablet_service.cc:1467] Processing CreateTablet for tablet 57b89b853b394680841eefdfa39077c9 (DEFAULT_TABLE table=test-table [id=118d770da198479b81b3c8285a2565f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:57.158717 29537 tablet_service.cc:1467] Processing CreateTablet for tablet 57b89b853b394680841eefdfa39077c9 (DEFAULT_TABLE table=test-table [id=118d770da198479b81b3c8285a2565f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:50:57.159578 29405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b89b853b394680841eefdfa39077c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:57.159766 29537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b89b853b394680841eefdfa39077c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:50:57.163111 29363 tablet_bootstrap.cc:492] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: Bootstrap starting.
I20260619 06:50:57.165444 29363 tablet_bootstrap.cc:654] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:57.166095 29616 tablet_bootstrap.cc:492] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: Bootstrap starting.
I20260619 06:50:57.166256 29363 log.cc:824] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:57.168149 29363 tablet_bootstrap.cc:492] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: No bootstrap required, opened a new log
I20260619 06:50:57.168349 29363 ts_tablet_manager.cc:1397] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:50:57.168618 29616 tablet_bootstrap.cc:654] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: Neither blocks nor log segments found. Creating new log.
I20260619 06:50:57.169369 29616 log.cc:824] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:57.171168 29616 tablet_bootstrap.cc:492] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: No bootstrap required, opened a new log
I20260619 06:50:57.171344 29616 ts_tablet_manager.cc:1397] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:50:57.172169 29363 raft_consensus.cc:348] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.172426 29363 raft_consensus.cc:374] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:57.172485 29363 raft_consensus.cc:729] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b437bff093e54a8b8ec74a1576770cb0, State: Initialized, Role: FOLLOWER
I20260619 06:50:57.172945 29363 consensus_queue.cc:260] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.173564 29471 heartbeater.cc:503] Master 127.14.116.254:46023 was elected leader, sending a full tablet report...
I20260619 06:50:57.173928 29363 ts_tablet_manager.cc:1428] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:50:57.174974 29616 raft_consensus.cc:348] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.175156 29616 raft_consensus.cc:374] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:50:57.175215 29616 raft_consensus.cc:729] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bac9c5cf17a4ddca7aaab6c527435d3, State: Initialized, Role: FOLLOWER
I20260619 06:50:57.175660 29616 consensus_queue.cc:260] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.176273 29603 heartbeater.cc:503] Master 127.14.116.254:46023 was elected leader, sending a full tablet report...
I20260619 06:50:57.176599 29616 ts_tablet_manager.cc:1428] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 06:50:57.184094 29472 tablet.cc:2369] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:50:57.235445 29333 tablet.cc:2369] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:50:57.267951 29620 raft_consensus.cc:482] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:50:57.268154 29620 raft_consensus.cc:504] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.269349 29620 leader_election.cc:288] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251), 22c8af2d6502469195259a79df948e46 (127.14.116.193:38417)
I20260619 06:50:57.274294 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.269547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:35251 (local address 127.14.116.195:51499)
0619 06:50:57.269676 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.269691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:57.269783 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:57.270178 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:57.270180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:57.270194 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:57.270315 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.270321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.271310 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.271312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.272812 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.272823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.273674 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.273678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.273868 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.273982 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 06:50:57.274058 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:50:57.274139 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":248}
I20260619 06:50:57.274354 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.269733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:38517 (local address 127.14.116.193:38417)
0619 06:50:57.269969 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:57.269973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:57.270039 (+ 66us) server_negotiation.cc:407] Connection header received
0619 06:50:57.270146 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:57.270149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:57.270208 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:57.270303 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:57.270574 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.271321 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.272932 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.273594 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.273700 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.273843 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:57.274112 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 06:50:57.274195 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260619 06:50:57.274696 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.269763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:51499 (local address 127.14.116.194:35251)
0619 06:50:57.269906 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:57.269910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:57.269924 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:50:57.269976 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:57.269980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:57.270060 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:57.270155 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:50:57.270436 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.271210 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.272932 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.273537 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.273678 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.273735 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:57.274495 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 06:50:57.274579 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:50:57.274775 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.269624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:38417 (local address 127.14.116.195:38517)
0619 06:50:57.269940 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.269951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:57.270053 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:57.270329 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:57.270331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:57.270347 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:50:57.270458 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.270469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.271418 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.271421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.272812 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:57.272823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.273691 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.273694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:57.273868 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.273979 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 06:50:57.274523 (+ 544us) client_negotiation.cc:240] Negotiation successful
0619 06:50:57.274590 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":85,"thread_start_us":96,"threads_started":1}
I20260619 06:50:57.275436 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b89b853b394680841eefdfa39077c9" candidate_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b437bff093e54a8b8ec74a1576770cb0" is_pre_election: true
I20260619 06:50:57.275421 29286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b89b853b394680841eefdfa39077c9" candidate_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c8af2d6502469195259a79df948e46" is_pre_election: true
I20260619 06:50:57.275777 29425 raft_consensus.cc:2454] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bac9c5cf17a4ddca7aaab6c527435d3 in term 0.
I20260619 06:50:57.275758 29286 raft_consensus.cc:2454] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bac9c5cf17a4ddca7aaab6c527435d3 in term 0.
I20260619 06:50:57.276252 29490 leader_election.cc:302] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22c8af2d6502469195259a79df948e46, 3bac9c5cf17a4ddca7aaab6c527435d3, b437bff093e54a8b8ec74a1576770cb0; no voters:
I20260619 06:50:57.276507 29620 raft_consensus.cc:2781] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:50:57.276602 29620 raft_consensus.cc:482] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:50:57.276659 29620 raft_consensus.cc:3037] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:57.277680 29620 raft_consensus.cc:504] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.278066 29620 leader_election.cc:288] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [CANDIDATE]: Term 1 election: Requested vote from peers b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251), 22c8af2d6502469195259a79df948e46 (127.14.116.193:38417)
I20260619 06:50:57.278434 29286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b89b853b394680841eefdfa39077c9" candidate_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c8af2d6502469195259a79df948e46"
I20260619 06:50:57.278463 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b89b853b394680841eefdfa39077c9" candidate_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b437bff093e54a8b8ec74a1576770cb0"
I20260619 06:50:57.278537 29286 raft_consensus.cc:3037] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:57.278592 29425 raft_consensus.cc:3037] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:50:57.279551 29286 raft_consensus.cc:2454] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bac9c5cf17a4ddca7aaab6c527435d3 in term 1.
I20260619 06:50:57.279659 29425 raft_consensus.cc:2454] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bac9c5cf17a4ddca7aaab6c527435d3 in term 1.
I20260619 06:50:57.279837 29490 leader_election.cc:302] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22c8af2d6502469195259a79df948e46, 3bac9c5cf17a4ddca7aaab6c527435d3; no voters:
I20260619 06:50:57.280062 29620 raft_consensus.cc:2781] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:50:57.280237 29620 raft_consensus.cc:686] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3bac9c5cf17a4ddca7aaab6c527435d3, State: Running, Role: LEADER
I20260619 06:50:57.280584 29620 consensus_queue.cc:237] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.283563 29144 catalog_manager.cc:5495] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195). New cstate: current_term: 1 leader_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } health_report { overall_health: UNKNOWN } } }
I20260619 06:50:57.311959 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.308992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.14.116.195:38043)
0619 06:50:57.309122 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:57.309126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:57.309138 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:50:57.309202 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:57.309205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:57.309255 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:57.309337 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:50:57.309528 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.310012 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.310747 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:57.310954 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:57.311025 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:57.311263 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:50:57.311673 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:50:57.311757 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:57.311815 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 06:50:57.311858 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:50:57.317346 29286 raft_consensus.cc:1261] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Refusing update from remote peer 3bac9c5cf17a4ddca7aaab6c527435d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:57.317346 29425 raft_consensus.cc:1261] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [term 1 FOLLOWER]: Refusing update from remote peer 3bac9c5cf17a4ddca7aaab6c527435d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:50:57.318105 29620 consensus_queue.cc:1035] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:57.318315 29622 consensus_queue.cc:1035] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:50:57.324973 29627 mvcc.cc:205] Tried to move back new op lower bound from 7298465207565189120 to 7298465207423979520. Current Snapshot: MvccSnapshot[applied={T|T < 7298465207565189120}]
I20260619 06:50:57.326195 29626 mvcc.cc:205] Tried to move back new op lower bound from 7298465207565189120 to 7298465207423979520. Current Snapshot: MvccSnapshot[applied={T|T < 7298465207565189120}]
I20260619 06:50:57.332100 29143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:50:57.332226 29143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:50:57.334400 29143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Sending DeleteTablet for 3 replicas of tablet 57b89b853b394680841eefdfa39077c9
I20260619 06:50:57.335018 29266 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b89b853b394680841eefdfa39077c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:57 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:40134
I20260619 06:50:57.335078 29405 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b89b853b394680841eefdfa39077c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:57 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:54832
I20260619 06:50:57.335129 29537 tablet_service.cc:1514] Processing DeleteTablet for tablet 57b89b853b394680841eefdfa39077c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:50:57 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:56802
I20260619 06:50:57.335333 29635 tablet_replica.cc:330] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: stopping tablet replica
I20260619 06:50:57.335470 29635 raft_consensus.cc:2229] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:50:57.335556 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 29182
I20260619 06:50:57.335621 29635 raft_consensus.cc:2258] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:57.336189 29636 tablet_replica.cc:330] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: stopping tablet replica
I20260619 06:50:57.336301 29637 tablet_replica.cc:330] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: stopping tablet replica
I20260619 06:50:57.336562 29636 raft_consensus.cc:2229] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:50:57.336591 29637 raft_consensus.cc:2229] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:50:57.336865 29637 raft_consensus.cc:2258] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:57.337037 29636 raft_consensus.cc:2258] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:50:57.338415 29637 ts_tablet_manager.cc:1905] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:57.338632 29636 ts_tablet_manager.cc:1905] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:50:57.341320 29636 ts_tablet_manager.cc:1918] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:57.341482 29636 log.cc:1196] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/57b89b853b394680841eefdfa39077c9
I20260619 06:50:57.341640 29637 ts_tablet_manager.cc:1918] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:50:57.341737 29637 log.cc:1196] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/57b89b853b394680841eefdfa39077c9
I20260619 06:50:57.342036 29636 ts_tablet_manager.cc:1939] T 57b89b853b394680841eefdfa39077c9 P 3bac9c5cf17a4ddca7aaab6c527435d3: Deleting consensus metadata
I20260619 06:50:57.342060 29637 ts_tablet_manager.cc:1939] T 57b89b853b394680841eefdfa39077c9 P b437bff093e54a8b8ec74a1576770cb0: Deleting consensus metadata
I20260619 06:50:57.343173 29129 catalog_manager.cc:4841] TS b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251): tablet 57b89b853b394680841eefdfa39077c9 (table test-table [id=118d770da198479b81b3c8285a2565f0]) successfully deleted
I20260619 06:50:57.343350 29129 catalog_manager.cc:4841] TS 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043): tablet 57b89b853b394680841eefdfa39077c9 (table test-table [id=118d770da198479b81b3c8285a2565f0]) successfully deleted
W20260619 06:50:57.343897 29128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:38417 (error 108)
I20260619 06:50:57.343950 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 29337
I20260619 06:50:57.345858 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.345530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:38417 (local address 127.0.0.1:40146)
0619 06:50:57.345683 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.345760 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 06:50:57.346078 29128 catalog_manager.cc:4584] TS 22c8af2d6502469195259a79df948e46 (127.14.116.193:38417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57b89b853b394680841eefdfa39077c9: Network error: Client connection negotiation failed: client connection to 127.14.116.193:38417: connect: Connection refused (error 111)
I20260619 06:50:57.350165 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 29474
I20260619 06:50:57.355687 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 29111
I20260619 06:50:57.361523 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46023
--webserver_interface=127.14.116.254
--webserver_port=35471
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:57.467495 29639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:57.467823 29639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:57.467901 29639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:57.471874 29639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:50:57.471946 29639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:57.471989 29639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:57.472030 29639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:50:57.472064 29639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:50:57.477166 29639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=35471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:57.478093 29639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:57.479053 29639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:57.487146 29644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:57.487196 29645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:57.487185 29639 server_base.cc:1034] running on GCE node
W20260619 06:50:57.487149 29647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:57.487950 29639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:57.488979 29639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:57.490131 29639 hybrid_clock.cc:648] HybridClock initialized: now 1781851857490119 us; error 25 us; skew 500 ppm
Jun 19 06:50:57 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851857, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:57.492885 29639 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:50:57.493687 29639 webserver.cc:460] Webserver started at http://127.14.116.254:35471/ using document root <none> and password file <none>
I20260619 06:50:57.494213 29639 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:57.494288 29639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:57.497968 29639 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:50:57.499993 29654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:57.501132 29639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:50:57.501259 29639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "a89f99fdd4c1470dbaa49ea9a365512e"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5cf004bc386a4c770e7aa3362370927a"
server_key_iv: "5cbb0205efb4ad1fb6f12df34b37ea69"
server_key_version: "encryptionkey@0"
I20260619 06:50:57.501751 29639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:57.515872 29639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:57.518934 29639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:57.519129 29639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:57.525574 29639 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:46023
I20260619 06:50:57.525853 29706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:46023 every 8 connection(s)
I20260619 06:50:57.526652 29639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:50:57.527274 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 29639
I20260619 06:50:57.527776 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:38417
--local_ip_for_outbound_sockets=127.14.116.193
--tserver_master_addrs=127.14.116.254:46023
--webserver_port=46251
--webserver_interface=127.14.116.193
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:50:57.531555 29707 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:50:57.533072 29707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Bootstrap starting.
I20260619 06:50:57.549508 29707 log.cc:824] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:57.559937 29707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:50:57.560287 29707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e: Bootstrap complete.
I20260619 06:50:57.564251 29707 raft_consensus.cc:348] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:57.564636 29707 raft_consensus.cc:729] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89f99fdd4c1470dbaa49ea9a365512e, State: Initialized, Role: FOLLOWER
I20260619 06:50:57.565109 29707 consensus_queue.cc:260] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:57.565245 29707 raft_consensus.cc:388] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:50:57.565332 29707 raft_consensus.cc:482] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:50:57.565451 29707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:50:57.567167 29707 raft_consensus.cc:504] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:57.567512 29707 leader_election.cc:302] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a89f99fdd4c1470dbaa49ea9a365512e; no voters:
I20260619 06:50:57.567857 29707 leader_election.cc:288] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:50:57.567936 29711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:50:57.568111 29711 raft_consensus.cc:686] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [term 3 LEADER]: Becoming Leader. State: Replica: a89f99fdd4c1470dbaa49ea9a365512e, State: Running, Role: LEADER
I20260619 06:50:57.568374 29711 consensus_queue.cc:237] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } }
I20260619 06:50:57.569042 29707 sys_catalog.cc:564] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:50:57.569255 29712 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } } }
I20260619 06:50:57.569268 29711 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a89f99fdd4c1470dbaa49ea9a365512e. Latest consensus state: current_term: 3 leader_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89f99fdd4c1470dbaa49ea9a365512e" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46023 } } }
I20260619 06:50:57.569610 29711 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:57.569571 29712 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89f99fdd4c1470dbaa49ea9a365512e [sys.catalog]: This master's current role is: LEADER
I20260619 06:50:57.569983 29715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:50:57.575079 29715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:50:57.575733 29715 catalog_manager.cc:1259] Loaded cluster ID: 01da7e66ac4f42d0806f18639afec59c
I20260619 06:50:57.575791 29715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 06:50:57.576783 29715 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1bcaa489ad google::LogMessage::Fail() at ??:0
@ 0x7f1bcaa4b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1bcaa484de google::LogMessage::Flush() at ??:0
@ 0x7f1bcaa49a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1bd5387fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1bd5388d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1bd5385579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1bd53c9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1bccc9a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1bccc9ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1bccc9c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1bccc8b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1bce2cf6db start_thread at ??:0
@ 0x7f1bc954661f clone at ??:0
W20260619 06:50:57.650063 29709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:57.650266 29709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:57.650307 29709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:57.654446 29709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:57.654525 29709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:57.654647 29709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:50:57.659788 29709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:38417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=46251
--tserver_master_addrs=127.14.116.254:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:57.660810 29709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:57.661798 29709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:57.671816 29751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:57.671957 29709 server_base.cc:1034] running on GCE node
W20260619 06:50:57.671821 29748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:57.671834 29749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:57.672546 29709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:57.673236 29709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:57.674444 29709 hybrid_clock.cc:648] HybridClock initialized: now 1781851857674413 us; error 39 us; skew 500 ppm
Jun 19 06:50:57 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851857, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:57.677636 29709 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:50:57.678606 29709 webserver.cc:460] Webserver started at http://127.14.116.193:46251/ using document root <none> and password file <none>
I20260619 06:50:57.679236 29709 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:57.679323 29709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:57.684103 29709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:50:57.686579 29758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:57.688047 29709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:50:57.688154 29709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "22c8af2d6502469195259a79df948e46"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "a6db6810da1270945c0e8f161f26b17e"
server_key_iv: "cacc05ef4296604e783998d0242e5886"
server_key_version: "encryptionkey@0"
I20260619 06:50:57.688854 29709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:57.719254 29709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:57.722980 29709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:57.723181 29709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:57.723951 29709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:57.725214 29765 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 06:50:57.729303 29709 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:50:57.729383 29709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260619 06:50:57.729445 29709 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:50:57.731554 29709 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:50:57.731631 29709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 06:50:57.731951 29765 tablet_bootstrap.cc:492] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Bootstrap starting.
I20260619 06:50:57.742008 29709 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:38417
I20260619 06:50:57.742233 29872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:38417 every 8 connection(s)
I20260619 06:50:57.743184 29709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:50:57.744235 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 29709
I20260619 06:50:57.744704 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:35251
--local_ip_for_outbound_sockets=127.14.116.194
--tserver_master_addrs=127.14.116.254:46023
--webserver_port=39731
--webserver_interface=127.14.116.194
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:50:57.753031 29765 log.cc:824] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Log is configured to *not* fsync() on all Append() calls
I20260619 06:50:57.761866 29765 tablet_bootstrap.cc:492] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:50:57.762257 29765 tablet_bootstrap.cc:492] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Bootstrap complete.
I20260619 06:50:57.763671 29765 ts_tablet_manager.cc:1397] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.013s
I20260619 06:50:57.766929 29765 raft_consensus.cc:348] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.767457 29765 raft_consensus.cc:729] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c8af2d6502469195259a79df948e46, State: Initialized, Role: FOLLOWER
I20260619 06:50:57.767990 29765 consensus_queue.cc:260] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:57.768993 29765 ts_tablet_manager.cc:1428] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 06:50:57.804265 29876 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:50:57.747745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:57593)
0619 06:50:57.748205 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.748256 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:57.749437 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:57.803315 (+ 53878us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":133,"threads_started":1}
W20260619 06:50:57.804829 29873 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:46023 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.254:46023: Client connection negotiation failed: client connection to 127.14.116.254:46023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:50:57.805529 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.805268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:33703)
0619 06:50:57.805399 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.805467 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 06:50:57.806190 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.805958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:43885)
0619 06:50:57.806066 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.806119 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:50:57.806370 29873 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:50:57.856778 29875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:57.857008 29875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:57.857048 29875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:57.861055 29875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:57.861111 29875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:57.861197 29875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:50:57.866225 29875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:35251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=39731
--tserver_master_addrs=127.14.116.254:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:57.867194 29875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:57.868214 29875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:57.877291 29883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:57.877300 29884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:57.877359 29875 server_base.cc:1034] running on GCE node
W20260619 06:50:57.877311 29886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:57.877985 29875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:57.878556 29875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:57.879678 29875 hybrid_clock.cc:648] HybridClock initialized: now 1781851857879644 us; error 41 us; skew 500 ppm
Jun 19 06:50:57 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851857, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:57.882331 29875 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:50:57.883121 29875 webserver.cc:460] Webserver started at http://127.14.116.194:39731/ using document root <none> and password file <none>
I20260619 06:50:57.883751 29875 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:57.883845 29875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:57.887387 29875 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:50:57.889539 29893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:57.890674 29875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:50:57.890799 29875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "b437bff093e54a8b8ec74a1576770cb0"
format_stamp: "Formatted at 2026-06-19 06:50:55 on dist-test-slave-9x3b"
server_key: "5d02f2ca0fadec036eb91fc567b44993"
server_key_iv: "3ed3dcf7b0fc68fccbf749346c8a25f7"
server_key_version: "encryptionkey@0"
I20260619 06:50:57.891316 29875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:57.903301 29875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:57.905850 29875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:57.906047 29875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:57.906582 29875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:57.907333 29875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:57.907402 29875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:57.907461 29875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:57.907511 29875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:57.915989 29875 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:35251
I20260619 06:50:57.916054 30006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:35251 every 8 connection(s)
I20260619 06:50:57.916951 29875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:50:57.920439 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.918917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:49051)
0619 06:50:57.919367 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.919778 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
W20260619 06:50:57.920856 30007 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:46023 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.254:46023: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
I20260619 06:50:57.921352 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.921169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:45417)
0619 06:50:57.921261 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.921304 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 06:50:57.921895 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:57.921723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:47837)
0619 06:50:57.921813 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:57.921850 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 06:50:57.922065 30007 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:50:57.922690 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 29875
I20260619 06:50:57.923032 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:38043
--local_ip_for_outbound_sockets=127.14.116.195
--tserver_master_addrs=127.14.116.254:46023
--webserver_port=43375
--webserver_interface=127.14.116.195
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:50:58.029290 30010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:50:58.029533 30010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:50:58.029575 30010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:50:58.033447 30010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:50:58.033492 30010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:50:58.033547 30010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:50:58.038582 30010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:38043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=43375
--tserver_master_addrs=127.14.116.254:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:50:58.039681 30010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:50:58.040758 30010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:50:58.049887 30018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:58.049875 30015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:50:58.049875 30016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:50:58.050194 30010 server_base.cc:1034] running on GCE node
I20260619 06:50:58.050614 30010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:50:58.051211 30010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:50:58.052397 30010 hybrid_clock.cc:648] HybridClock initialized: now 1781851858052382 us; error 24 us; skew 500 ppm
Jun 19 06:50:58 dist-test-slave-9x3b krb5kdc[28563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851858, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:50:58.054873 30010 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:50:58.055766 30010 webserver.cc:460] Webserver started at http://127.14.116.195:43375/ using document root <none> and password file <none>
I20260619 06:50:58.056249 30010 fs_manager.cc:359] Metadata directory not provided
I20260619 06:50:58.056316 30010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:50:58.059845 30010 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 06:50:58.061797 30025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:58.062938 30010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:50:58.063051 30010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "3bac9c5cf17a4ddca7aaab6c527435d3"
format_stamp: "Formatted at 2026-06-19 06:50:56 on dist-test-slave-9x3b"
server_key: "6612e51372b03974e7ce5950b2a86534"
server_key_iv: "63eaee314e91c14db82bc8328b9640c3"
server_key_version: "encryptionkey@0"
I20260619 06:50:58.063546 30010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:50:58.073861 30010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:50:58.076503 30010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:50:58.076681 30010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:50:58.077207 30010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:50:58.077961 30010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:50:58.078006 30010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:58.078071 30010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:50:58.078109 30010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:50:58.086675 30010 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:38043
I20260619 06:50:58.086699 30138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:38043 every 8 connection(s)
I20260619 06:50:58.087596 30010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:50:58.088820 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 30010
I20260619 06:50:58.091259 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:58.089652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:46109)
0619 06:50:58.090175 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:58.090606 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":144,"threads_started":1}
W20260619 06:50:58.091779 30139 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:46023 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.254:46023: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
I20260619 06:50:58.092291 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:58.092106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:57555)
0619 06:50:58.092201 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:58.092243 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 06:50:58.092831 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:58.092658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:60097)
0619 06:50:58.092747 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:58.092785 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 06:50:58.093011 30139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:50:58.807463 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:58.806965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:43841)
0619 06:50:58.807275 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:58.807329 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":68,"threads_started":1}
I20260619 06:50:58.922901 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:58.922543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:41879)
0619 06:50:58.922758 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:58.922808 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
I20260619 06:50:59.093861 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:59.093516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:45117)
0619 06:50:59.093722 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:59.093772 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
I20260619 06:50:59.464031 30146 raft_consensus.cc:482] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:50:59.464222 30146 raft_consensus.cc:504] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:50:59.465224 30146 leader_election.cc:288] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043), b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251)
Jun 19 06:50:59 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851857, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.195@KRBTEST.COM
Jun 19 06:50:59 dist-test-slave-9x3b krb5kdc[28563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851857, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.194@KRBTEST.COM
I20260619 06:50:59.478417 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:59.466025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:51571 (local address 127.14.116.194:35251)
0619 06:50:59.466317 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:59.466331 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:59.466366 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:50:59.467042 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:59.467049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:59.467360 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:59.467693 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:59.468730 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:59.469639 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:59.470780 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:59.471294 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:59.471456 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:59.474479 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:59.474488 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:59.474502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:59.474534 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:59.476294 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:59.476729 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:59.476734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:59.476739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:59.476845 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:59.477123 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:59.477126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:59.477128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:59.477456 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:59.477588 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:59.478027 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 06:50:59.478263 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":107,"threads_started":1}
I20260619 06:50:59.478406 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:59.465822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:35251 (local address 127.14.116.193:51571)
0619 06:50:59.466094 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:59.466108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:59.466201 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:59.467670 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:59.467673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:59.468109 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:59.468573 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:59.468588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:59.469791 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:59.469798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:59.470661 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:59.470670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:59.471599 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:59.471602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:59.471722 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:59.472189 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:59.472206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:59.474267 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:59.476415 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:59.476418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:59.476420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:59.476618 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:59.476946 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:59.476949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:59.476951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:59.477031 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:59.477559 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:59.477562 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:59.477750 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 06:50:59.477922 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 06:50:59.478220 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":116,"spinlock_wait_cycles":5760,"thread_start_us":72,"threads_started":1}
I20260619 06:50:59.478552 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:59.465453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:38043 (local address 127.14.116.193:41529)
0619 06:50:59.465684 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:59.465698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:50:59.465793 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:50:59.467513 (+ 1720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:50:59.467522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:50:59.468111 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:50:59.468573 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:59.468587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:59.469792 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:59.469796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:59.470661 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:50:59.470670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:50:59.471366 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:50:59.471370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:50:59.471527 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:59.472112 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:50:59.472139 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:50:59.473875 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:50:59.476230 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:59.476236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:59.476248 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:59.476483 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:59.476825 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:50:59.476828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:50:59.476830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:50:59.476956 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:50:59.477535 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:50:59.477540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:50:59.477754 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 06:50:59.477922 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 06:50:59.478304 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":226,"spinlock_wait_cycles":28288,"thread_start_us":98,"threads_started":1}
I20260619 06:50:59.479014 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:59.465557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:41529 (local address 127.14.116.195:38043)
0619 06:50:59.465908 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 06:50:59.465937 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 06:50:59.465985 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:50:59.466810 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:50:59.466817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:50:59.467150 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:50:59.467389 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:50:59.468737 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:59.469629 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:59.470778 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:50:59.471234 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:50:59.471419 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:50:59.474123 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:50:59.474128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:50:59.474139 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:50:59.474164 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:50:59.476110 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:59.476599 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:59.476611 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:59.476614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:59.476717 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:50:59.477051 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:50:59.477055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:50:59.477057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:50:59.477420 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:50:59.477560 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:50:59.478031 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 06:50:59.478304 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":54,"threads_started":1}
W20260619 06:50:59.480873 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
W20260619 06:50:59.481073 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
I20260619 06:50:59.481155 29760 leader_election.cc:302] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22c8af2d6502469195259a79df948e46; no voters: 3bac9c5cf17a4ddca7aaab6c527435d3, b437bff093e54a8b8ec74a1576770cb0
I20260619 06:50:59.481410 30146 raft_consensus.cc:2726] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:50:59.808429 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:59.808169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:36989)
0619 06:50:59.808294 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:59.808346 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 06:50:59.923997 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:50:59.923579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:56763)
0619 06:50:59.923845 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 06:50:59.923902 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
I20260619 06:51:00.094933 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:00.094580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:44403)
0619 06:51:00.094800 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:00.094849 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":68,"threads_started":1}
I20260619 06:51:00.809540 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:00.809158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:38137)
0619 06:51:00.809382 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:00.809433 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260619 06:51:00.925097 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:00.924689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:48331)
0619 06:51:00.924954 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:00.925011 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260619 06:51:01.096011 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:01.095558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:52321)
0619 06:51:01.095849 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:01.095913 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
I20260619 06:51:01.218660 30156 raft_consensus.cc:482] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:51:01.218824 30156 raft_consensus.cc:504] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:51:01.219326 30156 leader_election.cc:288] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043), b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251)
W20260619 06:51:01.220198 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
W20260619 06:51:01.220360 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
I20260619 06:51:01.220422 29760 leader_election.cc:302] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22c8af2d6502469195259a79df948e46; no voters: 3bac9c5cf17a4ddca7aaab6c527435d3, b437bff093e54a8b8ec74a1576770cb0
I20260619 06:51:01.220613 30156 raft_consensus.cc:2726] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:51:01.810709 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:01.810295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:53661)
0619 06:51:01.810561 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:01.810616 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260619 06:51:01.926193 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:01.925791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:43085)
0619 06:51:01.926035 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:01.926089 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
I20260619 06:51:02.097023 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:02.096651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:55629)
0619 06:51:02.096885 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:02.096941 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
I20260619 06:51:02.812003 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:02.811566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:38803)
0619 06:51:02.811861 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:02.811912 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":70,"threads_started":1}
I20260619 06:51:02.927388 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:02.926947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:43975)
0619 06:51:02.927236 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:02.927303 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":118,"threads_started":1}
I20260619 06:51:03.098106 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:03.097727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:59875)
0619 06:51:03.097962 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:03.098015 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260619 06:51:03.543844 30163 raft_consensus.cc:482] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:51:03.543964 30163 raft_consensus.cc:504] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:51:03.544291 30163 leader_election.cc:288] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043), b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251)
W20260619 06:51:03.545161 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
W20260619 06:51:03.545334 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
I20260619 06:51:03.545411 29760 leader_election.cc:302] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22c8af2d6502469195259a79df948e46; no voters: 3bac9c5cf17a4ddca7aaab6c527435d3, b437bff093e54a8b8ec74a1576770cb0
I20260619 06:51:03.545570 30163 raft_consensus.cc:2726] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:51:03.813038 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:03.812653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:51897)
0619 06:51:03.812885 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:03.812945 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
I20260619 06:51:03.928414 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:03.928055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:44667)
0619 06:51:03.928283 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:03.928332 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260619 06:51:04.099165 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:04.098793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:35507)
0619 06:51:04.099031 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:04.099082 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260619 06:51:04.814082 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:04.813692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:53769)
0619 06:51:04.813935 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:04.813988 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260619 06:51:04.929483 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:04.929035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:38213)
0619 06:51:04.929328 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:04.929394 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
I20260619 06:51:05.100297 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:05.099858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:51039)
0619 06:51:05.100150 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:05.100203 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":77,"threads_started":1}
I20260619 06:51:05.815301 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:05.814902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:54927)
0619 06:51:05.815151 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:05.815206 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260619 06:51:05.930569 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:05.930147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:34749)
0619 06:51:05.930423 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:05.930481 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260619 06:51:06.101424 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:06.100979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:32827)
0619 06:51:06.101283 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:06.101338 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":69,"threads_started":1}
I20260619 06:51:06.816589 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:06.816078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:57555)
0619 06:51:06.816351 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:06.816413 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260619 06:51:06.931664 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:06.931237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:49023)
0619 06:51:06.931474 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:06.931526 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260619 06:51:07.102555 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:07.102142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:55791)
0619 06:51:07.102383 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:07.102447 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260619 06:51:07.817829 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:07.817462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:36379)
0619 06:51:07.817685 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:07.817736 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260619 06:51:07.877676 30177 raft_consensus.cc:482] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:51:07.877842 30177 raft_consensus.cc:504] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:51:07.878230 30177 leader_election.cc:288] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043), b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251)
W20260619 06:51:07.878995 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
W20260619 06:51:07.879138 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
I20260619 06:51:07.879199 29760 leader_election.cc:302] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22c8af2d6502469195259a79df948e46; no voters: 3bac9c5cf17a4ddca7aaab6c527435d3, b437bff093e54a8b8ec74a1576770cb0
I20260619 06:51:07.879371 30177 raft_consensus.cc:2726] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:51:07.932786 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:07.932399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:35119)
0619 06:51:07.932648 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:07.932700 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260619 06:51:08.103713 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:08.103269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:48079)
0619 06:51:08.103527 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:08.103576 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
I20260619 06:51:08.819186 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:08.818786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:42371)
0619 06:51:08.819032 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:08.819084 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
I20260619 06:51:08.933840 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:08.933480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:60053)
0619 06:51:08.933703 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:08.933755 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260619 06:51:09.104732 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:09.104357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:42959)
0619 06:51:09.104590 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:09.104643 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":64,"threads_started":1}
I20260619 06:51:09.820312 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:09.819930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:56773)
0619 06:51:09.820170 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:09.820221 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260619 06:51:09.934923 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:09.934553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:55983)
0619 06:51:09.934777 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:09.934834 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260619 06:51:10.105854 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:10.105465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:41199)
0619 06:51:10.105707 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:10.105764 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
I20260619 06:51:10.366385 30186 raft_consensus.cc:482] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:51:10.366515 30186 raft_consensus.cc:504] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bac9c5cf17a4ddca7aaab6c527435d3" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38043 } } peers { permanent_uuid: "b437bff093e54a8b8ec74a1576770cb0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 35251 } } peers { permanent_uuid: "22c8af2d6502469195259a79df948e46" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 38417 } }
I20260619 06:51:10.367040 30186 leader_election.cc:288] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043), b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251)
W20260619 06:51:10.367743 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3bac9c5cf17a4ddca7aaab6c527435d3 (127.14.116.195:38043): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
W20260619 06:51:10.367906 29760 leader_election.cc:341] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b437bff093e54a8b8ec74a1576770cb0 (127.14.116.194:35251): Not found: Tablet not found: 57b89b853b394680841eefdfa39077c9
I20260619 06:51:10.367965 29760 leader_election.cc:302] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22c8af2d6502469195259a79df948e46; no voters: 3bac9c5cf17a4ddca7aaab6c527435d3, b437bff093e54a8b8ec74a1576770cb0
I20260619 06:51:10.368121 30186 raft_consensus.cc:2726] T 57b89b853b394680841eefdfa39077c9 P 22c8af2d6502469195259a79df948e46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:51:10.821475 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:10.821060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:42495)
0619 06:51:10.821316 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:10.821370 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260619 06:51:10.936210 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:10.935813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:51551)
0619 06:51:10.936059 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:10.936116 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
I20260619 06:51:11.107055 30189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:11.106649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:59785)
0619 06:51:11.106908 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:11.106965 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260619 06:51:11.822551 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:11.822174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:56607)
0619 06:51:11.822408 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:11.822459 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260619 06:51:11.937400 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:11.936921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:52593)
0619 06:51:11.937253 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:11.937308 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":146,"threads_started":1}
I20260619 06:51:12.108240 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:12.107776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.195:50445)
0619 06:51:12.108069 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:12.108126 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260619 06:51:12.824013 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:12.823428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.193:53401)
0619 06:51:12.823813 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:12.823882 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":71,"threads_started":1}
I20260619 06:51:12.938621 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:12.938243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46023 (local address 127.14.116.194:54233)
0619 06:51:12.938476 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:12.938525 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260619 06:51:13.090700 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 29709
I20260619 06:51:13.096748 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 29875
I20260619 06:51:13.101733 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 30010
2026-06-19T06:51:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18982 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:51:13 dist-test-slave-9x3b krb5kdc[30197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:51:13 dist-test-slave-9x3b krb5kdc[30197](info): set up 2 sockets
Jun 19 06:51:13 dist-test-slave-9x3b krb5kdc[30197](info): commencing operation
krb5kdc: starting...
W20260619 06:51:14.147812 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:51:14 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:51:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:51:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:51:14.260854 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38983
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:38983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:51:14.365573 30210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:14.365820 30210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:14.365859 30210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:14.369827 30210 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:51:14.369876 30210 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:51:14.369892 30210 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:51:14.369905 30210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:51:14.369923 30210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:14.369941 30210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:14.369957 30210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:51:14.369972 30210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:51:14.375090 30210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:38983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:14.376101 30210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:14.377094 30210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:14.385583 30215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:14.385576 30216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:14.385576 30218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:14.385615 30210 server_base.cc:1034] running on GCE node
I20260619 06:51:14.386463 30210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:14.387554 30210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:14.388711 30210 hybrid_clock.cc:648] HybridClock initialized: now 1781851874388698 us; error 25 us; skew 500 ppm
Jun 19 06:51:14 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:14.391283 30210 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:51:14.392204 30210 webserver.cc:460] Webserver started at http://127.14.116.254:38425/ using document root <none> and password file <none>
I20260619 06:51:14.392745 30210 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:14.392835 30210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:14.393061 30210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:51:14.395027 30210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "4bf98da454e54cac98230611a5f4e7bf"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "55bfd8575994fd711d95cf82d362f4c7"
server_key_iv: "1ca7edf99a00a4a2574d12514fdbc3ab"
server_key_version: "encryptionkey@0"
I20260619 06:51:14.395573 30210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "4bf98da454e54cac98230611a5f4e7bf"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "55bfd8575994fd711d95cf82d362f4c7"
server_key_iv: "1ca7edf99a00a4a2574d12514fdbc3ab"
server_key_version: "encryptionkey@0"
I20260619 06:51:14.399063 30210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:51:14.401360 30225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:14.402475 30210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:51:14.402611 30210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "4bf98da454e54cac98230611a5f4e7bf"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "55bfd8575994fd711d95cf82d362f4c7"
server_key_iv: "1ca7edf99a00a4a2574d12514fdbc3ab"
server_key_version: "encryptionkey@0"
I20260619 06:51:14.402716 30210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:14.421413 30210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:14.424037 30210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:14.424229 30210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:14.430773 30210 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:38983
I20260619 06:51:14.431061 30277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:38983 every 8 connection(s)
I20260619 06:51:14.431730 30210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:51:14.434449 30278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:51:14.436522 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 30210
I20260619 06:51:14.436655 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:51:14.436939 14803 external_mini_cluster.cc:1428] Setting key 7f95f27d73bed75b37bfe5a8f948deed
I20260619 06:51:14.440184 30278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Bootstrap starting.
I20260619 06:51:14.442315 30278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Neither blocks nor log segments found. Creating new log.
I20260619 06:51:14.443024 30278 log.cc:824] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:51:14.444810 30278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: No bootstrap required, opened a new log
Jun 19 06:51:14 dist-test-slave-9x3b krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:51:14.447906 30278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:14.448117 30278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:51:14.448174 30278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf98da454e54cac98230611a5f4e7bf, State: Initialized, Role: FOLLOWER
I20260619 06:51:14.448617 30278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:14.448738 30278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:51:14.448818 30278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:51:14.448906 30278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:51:14.449704 30278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:14.450085 30278 leader_election.cc:302] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bf98da454e54cac98230611a5f4e7bf; no voters:
I20260619 06:51:14.450455 30278 leader_election.cc:288] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:51:14.450606 30283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:51:14.450785 30283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 1 LEADER]: Becoming Leader. State: Replica: 4bf98da454e54cac98230611a5f4e7bf, State: Running, Role: LEADER
I20260619 06:51:14.451126 30283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:14.451504 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:14.438262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.14.116.254:38983)
0619 06:51:14.438678 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:14.438691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:14.438725 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:51:14.439364 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:14.439380 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:14.439769 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:14.440103 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:14.440892 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.441834 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.442753 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.443253 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.443409 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:14.446166 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:14.446171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:14.446182 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:14.446206 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:14.448515 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:14.449039 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:14.449044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:14.449048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:14.449147 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:14.449440 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:14.449444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:14.449446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:14.449794 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:14.449979 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:14.450294 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 06:51:14.450600 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":141,"threads_started":1}
I20260619 06:51:14.451748 30278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:51:14.452297 30283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bf98da454e54cac98230611a5f4e7bf. Latest consensus state: current_term: 1 leader_uuid: "4bf98da454e54cac98230611a5f4e7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } } }
I20260619 06:51:14.452431 30283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:51:14.452695 30284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bf98da454e54cac98230611a5f4e7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } } }
I20260619 06:51:14.452802 30284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:51:14.453495 30288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:51:14.456317 30288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:51:14.461652 30288 catalog_manager.cc:1347] Generated new cluster ID: 68d8eaecdea44d079a0418f7376052f2
I20260619 06:51:14.461717 30288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:51:14.469993 30288 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:51:14.471148 30288 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:51:14.484454 30288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Generated new TSK 0
I20260619 06:51:14.485191 30288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:51:14.512147 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38983
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:51:14.620007 30304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:14.620258 30304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:14.620301 30304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:14.624326 30304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:14.624384 30304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:14.624464 30304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:51:14.629583 30304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:14.630589 30304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:14.631699 30304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:14.640767 30309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:14.640763 30310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:14.640808 30312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:14.640944 30304 server_base.cc:1034] running on GCE node
I20260619 06:51:14.641399 30304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:14.641981 30304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:14.643133 30304 hybrid_clock.cc:648] HybridClock initialized: now 1781851874643119 us; error 26 us; skew 500 ppm
Jun 19 06:51:14 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:14.645725 30304 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:51:14.646500 30304 webserver.cc:460] Webserver started at http://127.14.116.193:37923/ using document root <none> and password file <none>
I20260619 06:51:14.647081 30304 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:14.647141 30304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:14.647403 30304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:51:14.649369 30304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "6ba4fba1f7674725a525b657b93ee784"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "6874efaf4be6069ae19e71b0a56bac41"
server_key_iv: "2646798ff135f41bcaea26237b5241f2"
server_key_version: "encryptionkey@0"
I20260619 06:51:14.649920 30304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "6ba4fba1f7674725a525b657b93ee784"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "6874efaf4be6069ae19e71b0a56bac41"
server_key_iv: "2646798ff135f41bcaea26237b5241f2"
server_key_version: "encryptionkey@0"
I20260619 06:51:14.653481 30304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:51:14.655799 30319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:14.656965 30304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:51:14.657106 30304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "6ba4fba1f7674725a525b657b93ee784"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "6874efaf4be6069ae19e71b0a56bac41"
server_key_iv: "2646798ff135f41bcaea26237b5241f2"
server_key_version: "encryptionkey@0"
I20260619 06:51:14.657193 30304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:14.676508 30304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:14.679107 30304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:14.679255 30304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:14.679795 30304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:51:14.680509 30304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:51:14.680553 30304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:14.680589 30304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:51:14.680612 30304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:14.688881 30304 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:44701
I20260619 06:51:14.688952 30432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:44701 every 8 connection(s)
I20260619 06:51:14.689726 30304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:51:14.697851 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 30304
I20260619 06:51:14.697978 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:51:14.698266 14803 external_mini_cluster.cc:1428] Setting key 425ec58561cc2cb0cbb45b9a8f41866b
Jun 19 06:51:14 dist-test-slave-9x3b krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:51:14.702932 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:14.691432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:57601 (local address 127.14.116.254:38983)
0619 06:51:14.691577 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:14.691582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:14.692668 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 06:51:14.693459 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:14.693462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:14.693504 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:14.693578 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:14.694690 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.695230 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.696356 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.696592 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.696672 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:14.699034 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:14.699037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:14.699039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:14.699062 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:14.700789 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:14.701301 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:14.701307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:14.701310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:14.701420 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:14.701813 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:14.701815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:14.701816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:14.701970 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:14.702044 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:14.702649 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 06:51:14.702767 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:51:14.703969 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:14.691877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:57601)
0619 06:51:14.692519 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:14.692559 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:14.693276 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:14.693731 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:14.693737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:14.694074 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:51:14.694551 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:14.694564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.695332 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.695335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:14.696247 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:14.696256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.696719 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.696722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:14.696849 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:14.697356 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:51:14.697371 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:51:14.698867 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:51:14.700913 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:14.700918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:14.700929 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:14.701171 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:14.701527 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:14.701531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:14.701535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:14.701719 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:14.702070 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:51:14.702079 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:51:14.702387 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 06:51:14.702603 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 06:51:14.702897 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":120,"threads_started":1}
I20260619 06:51:14.705211 30433 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38983
I20260619 06:51:14.705470 30433 heartbeater.cc:464] Registering TS with master...
I20260619 06:51:14.706085 30433 heartbeater.cc:511] Master 127.14.116.254:38983 requested a full tablet report, sending...
I20260619 06:51:14.707854 30242 ts_manager.cc:194] Registered new tserver with Master: 6ba4fba1f7674725a525b657b93ee784 (127.14.116.193:44701)
I20260619 06:51:14.709471 30242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:57601
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:51:14.727576 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38983
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:51:14.833642 30438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:14.833868 30438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:14.833935 30438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:14.837921 30438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:14.837967 30438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:14.838056 30438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:51:14.842978 30438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:14.843927 30438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:14.844846 30438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:14.853941 30443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:14.853931 30446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:14.853931 30444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:14.854003 30438 server_base.cc:1034] running on GCE node
I20260619 06:51:14.854522 30438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:14.855130 30438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:14.856280 30438 hybrid_clock.cc:648] HybridClock initialized: now 1781851874856268 us; error 24 us; skew 500 ppm
Jun 19 06:51:14 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:14.858981 30438 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:51:14.859828 30438 webserver.cc:460] Webserver started at http://127.14.116.194:38823/ using document root <none> and password file <none>
I20260619 06:51:14.860350 30438 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:14.860409 30438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:14.860672 30438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:51:14.862550 30438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "0240b0c340354a69b5dc5d08bfd16fe6"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "fc6a4ec3ae5da2948a03816d36d2e1cb"
server_key_iv: "e0550648603b72796e7eaab503f040d3"
server_key_version: "encryptionkey@0"
I20260619 06:51:14.863121 30438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "0240b0c340354a69b5dc5d08bfd16fe6"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "fc6a4ec3ae5da2948a03816d36d2e1cb"
server_key_iv: "e0550648603b72796e7eaab503f040d3"
server_key_version: "encryptionkey@0"
I20260619 06:51:14.866528 30438 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:51:14.868778 30453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:14.869891 30438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:51:14.870016 30438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "0240b0c340354a69b5dc5d08bfd16fe6"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "fc6a4ec3ae5da2948a03816d36d2e1cb"
server_key_iv: "e0550648603b72796e7eaab503f040d3"
server_key_version: "encryptionkey@0"
I20260619 06:51:14.870112 30438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:14.887842 30438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:14.891024 30438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:14.891224 30438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:14.891825 30438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:51:14.892621 30438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:51:14.892668 30438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:14.892736 30438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:51:14.892777 30438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:14.901079 30438 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:36367
I20260619 06:51:14.901105 30566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:36367 every 8 connection(s)
I20260619 06:51:14.901908 30438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:51:14.903352 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 30438
I20260619 06:51:14.903497 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:51:14.903854 14803 external_mini_cluster.cc:1428] Setting key d64064e9847788bea029ab471cf8cbe1
Jun 19 06:51:14 dist-test-slave-9x3b krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851874, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:51:14.915370 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:14.903591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:58133 (local address 127.14.116.254:38983)
0619 06:51:14.903748 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:14.903753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:14.904991 (+ 1238us) server_negotiation.cc:407] Connection header received
0619 06:51:14.906005 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:14.906011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:14.906073 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:14.906164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:14.907292 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.907900 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.909004 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.909230 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.909349 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:14.912056 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:14.912059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:14.912062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:14.912089 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:14.913546 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:14.913988 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:14.913990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:14.913992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:14.914099 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:14.914417 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:14.914420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:14.914421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:14.914586 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:14.914663 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:14.915095 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 06:51:14.915193 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:51:14.916221 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:14.904077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:58133)
0619 06:51:14.904830 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:14.904893 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:14.905770 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:14.906313 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:14.906320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:14.906649 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:51:14.907145 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:14.907172 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.908021 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.908024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:14.908877 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:14.908886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:14.909371 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:14.909376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:14.909540 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:14.910138 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:51:14.910153 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:51:14.911897 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:51:14.913648 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:14.913652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:14.913663 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:14.913904 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:14.914199 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:14.914202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:14.914203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:14.914337 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:14.914670 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:51:14.914675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:51:14.914891 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 06:51:14.915035 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 06:51:14.915263 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":133,"threads_started":1}
I20260619 06:51:14.917383 30567 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38983
I20260619 06:51:14.917619 30567 heartbeater.cc:464] Registering TS with master...
I20260619 06:51:14.918084 30567 heartbeater.cc:511] Master 127.14.116.254:38983 requested a full tablet report, sending...
I20260619 06:51:14.919088 30242 ts_manager.cc:194] Registered new tserver with Master: 0240b0c340354a69b5dc5d08bfd16fe6 (127.14.116.194:36367)
I20260619 06:51:14.919715 30242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:58133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:51:14.929548 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38983
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:51:15.036473 30572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:15.036731 30572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:15.036818 30572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:15.040931 30572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:15.041013 30572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:15.041095 30572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:51:15.046182 30572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:15.047197 30572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:15.048209 30572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:15.057716 30577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:15.057674 30578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:15.057770 30572 server_base.cc:1034] running on GCE node
W20260619 06:51:15.057685 30580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:15.058324 30572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:15.058856 30572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:15.060024 30572 hybrid_clock.cc:648] HybridClock initialized: now 1781851875060011 us; error 24 us; skew 500 ppm
Jun 19 06:51:15 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851875, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:15.063665 30572 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:51:15.064738 30572 webserver.cc:460] Webserver started at http://127.14.116.195:33727/ using document root <none> and password file <none>
I20260619 06:51:15.065395 30572 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:15.065460 30572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:15.065666 30572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:51:15.068290 30572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "1c7e8db0bfc2464d8694941a0b776da2"
format_stamp: "Formatted at 2026-06-19 06:51:15 on dist-test-slave-9x3b"
server_key: "3ecef243ed74ce300e236ad0a7372b7e"
server_key_iv: "a7493993a14a0ea48a21b96232557d1d"
server_key_version: "encryptionkey@0"
I20260619 06:51:15.068953 30572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "1c7e8db0bfc2464d8694941a0b776da2"
format_stamp: "Formatted at 2026-06-19 06:51:15 on dist-test-slave-9x3b"
server_key: "3ecef243ed74ce300e236ad0a7372b7e"
server_key_iv: "a7493993a14a0ea48a21b96232557d1d"
server_key_version: "encryptionkey@0"
I20260619 06:51:15.072923 30572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:51:15.075187 30587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:15.076377 30572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:51:15.076490 30572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "1c7e8db0bfc2464d8694941a0b776da2"
format_stamp: "Formatted at 2026-06-19 06:51:15 on dist-test-slave-9x3b"
server_key: "3ecef243ed74ce300e236ad0a7372b7e"
server_key_iv: "a7493993a14a0ea48a21b96232557d1d"
server_key_version: "encryptionkey@0"
I20260619 06:51:15.076584 30572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:15.094923 30572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:15.097896 30572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:15.098049 30572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:15.098608 30572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:51:15.099356 30572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:51:15.099409 30572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:15.099475 30572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:51:15.099515 30572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:15.107899 30572 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:43867
I20260619 06:51:15.107924 30700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:43867 every 8 connection(s)
I20260619 06:51:15.108754 30572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:51:15.114850 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 30572
I20260619 06:51:15.114984 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:51:15.115280 14803 external_mini_cluster.cc:1428] Setting key 14e4d869c75ee41a240940fa8d1d0154
Jun 19 06:51:15 dist-test-slave-9x3b krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851875, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:51:15.121615 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.110385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:55191 (local address 127.14.116.254:38983)
0619 06:51:15.110522 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:15.110526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:15.111483 (+ 957us) server_negotiation.cc:407] Connection header received
0619 06:51:15.112389 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:15.112393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:15.112449 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:15.112563 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:15.113643 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.114173 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.115799 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.116088 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.116213 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.118660 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:15.118663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:15.118665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:15.118689 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:15.119998 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:15.120430 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:15.120433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:15.120434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:15.120509 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:15.120774 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:15.120776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:15.120777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:15.120932 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:15.121009 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:15.121405 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 06:51:15.121504 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:51:15.122345 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.110732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:55191)
0619 06:51:15.111337 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:15.111373 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:15.112194 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:15.112725 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:15.112731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:15.113070 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:51:15.113505 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.113517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.114315 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.114319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.115676 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.115692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.116244 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.116248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.116435 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.117046 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:51:15.117062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:51:15.118520 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:51:15.120095 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:15.120099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:15.120109 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:15.120339 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:15.120583 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:15.120585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:15.120586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:15.120701 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:15.121016 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:51:15.121021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:51:15.121220 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 06:51:15.121361 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:51:15.121588 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":106,"threads_started":1}
I20260619 06:51:15.123349 30701 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38983
I20260619 06:51:15.123575 30701 heartbeater.cc:464] Registering TS with master...
I20260619 06:51:15.124024 30701 heartbeater.cc:511] Master 127.14.116.254:38983 requested a full tablet report, sending...
I20260619 06:51:15.124941 30242 ts_manager.cc:194] Registered new tserver with Master: 1c7e8db0bfc2464d8694941a0b776da2 (127.14.116.195:43867)
I20260619 06:51:15.125440 30242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:55191
I20260619 06:51:15.128942 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:51:15.137048 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.130727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.14.116.254:38983)
0619 06:51:15.130837 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:15.130840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:15.131004 (+ 164us) server_negotiation.cc:407] Connection header received
0619 06:51:15.131100 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:15.131102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:15.131136 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:15.131199 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:15.131859 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.132333 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.133092 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.133287 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.133357 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.134607 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:15.134610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:15.134612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:15.134630 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:15.135794 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:15.136155 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:15.136158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:15.136159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:15.136225 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:15.136422 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:15.136424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:15.136425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:15.136600 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:15.136699 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:15.136865 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 06:51:15.136946 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:51:15.139761 30242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:51:15.142620 30242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:51:15.156610 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.150476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:43867 (local address 127.0.0.1:46394)
0619 06:51:15.151453 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:15.151504 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:15.151726 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:15.152262 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:15.152268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:15.152333 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:51:15.152457 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.152464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.153785 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.153788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.155019 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.155029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.156070 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.156073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.156186 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.156223 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:51:15.156335 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:51:15.156456 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":101,"thread_start_us":71,"threads_started":1}
I20260619 06:51:15.156637 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.150595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.14.116.195:43867)
0619 06:51:15.151167 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:15.151182 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:15.151586 (+ 404us) server_negotiation.cc:407] Connection header received
0619 06:51:15.151885 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:15.151890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:15.152097 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:15.152274 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:51:15.152719 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.153612 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.155149 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.155946 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.156072 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.156194 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:15.156378 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 06:51:15.156483 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":119,"threads_started":1}
I20260619 06:51:15.157233 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.150848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:44701 (local address 127.0.0.1:39490)
0619 06:51:15.151971 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:15.151983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:15.152060 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:15.152548 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:15.152554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:15.152577 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:51:15.152714 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.152721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.154166 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.154170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.155642 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.155648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.156844 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.156849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.156954 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.156980 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:51:15.157048 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:51:15.157114 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":118,"thread_start_us":75,"threads_started":1}
I20260619 06:51:15.157325 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.151245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.14.116.193:44701)
0619 06:51:15.151656 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:15.151667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:15.152054 (+ 387us) server_negotiation.cc:407] Connection header received
0619 06:51:15.152179 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:15.152237 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:15.152413 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:15.152551 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:51:15.152840 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.154014 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.155782 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.156523 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.156638 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.156780 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:15.157088 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 06:51:15.157182 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":100,"threads_started":1}
I20260619 06:51:15.158005 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.150542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:36367 (local address 127.0.0.1:45886)
0619 06:51:15.151794 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:15.151807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:15.151899 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:15.152570 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:15.152573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:15.152593 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:51:15.152722 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.152727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.154166 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.154170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.155633 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.155644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.156883 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.156888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.157681 (+ 793us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.157704 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:51:15.157806 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:51:15.157874 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"mutex_wait_us":162,"thread_start_us":88,"threads_started":1}
I20260619 06:51:15.158066 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.151438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.14.116.194:36367)
0619 06:51:15.151822 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:15.151837 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:15.151860 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:51:15.152047 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:15.152053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:15.152246 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:15.152391 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:51:15.152901 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.154018 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.155768 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.156564 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.156687 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.156818 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:15.157837 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0619 06:51:15.157927 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":103,"threads_started":1}
I20260619 06:51:15.159385 30635 tablet_service.cc:1467] Processing CreateTablet for tablet c5926d463040437fb930f6c544f20449 (DEFAULT_TABLE table=test-table [id=3e4e1a4f451947e9a0c1b1882bb5d1f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:51:15.159947 30367 tablet_service.cc:1467] Processing CreateTablet for tablet c5926d463040437fb930f6c544f20449 (DEFAULT_TABLE table=test-table [id=3e4e1a4f451947e9a0c1b1882bb5d1f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:51:15.160511 30635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5926d463040437fb930f6c544f20449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:51:15.160934 30367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5926d463040437fb930f6c544f20449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:51:15.162220 30501 tablet_service.cc:1467] Processing CreateTablet for tablet c5926d463040437fb930f6c544f20449 (DEFAULT_TABLE table=test-table [id=3e4e1a4f451947e9a0c1b1882bb5d1f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:51:15.163131 30501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5926d463040437fb930f6c544f20449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:51:15.166333 30718 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: Bootstrap starting.
I20260619 06:51:15.167992 30719 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Bootstrap starting.
I20260619 06:51:15.168431 30718 tablet_bootstrap.cc:654] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: Neither blocks nor log segments found. Creating new log.
I20260619 06:51:15.169574 30720 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Bootstrap starting.
I20260619 06:51:15.169716 30718 log.cc:824] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:51:15.170195 30719 tablet_bootstrap.cc:654] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Neither blocks nor log segments found. Creating new log.
I20260619 06:51:15.171271 30719 log.cc:824] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Log is configured to *not* fsync() on all Append() calls
I20260619 06:51:15.171871 30720 tablet_bootstrap.cc:654] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Neither blocks nor log segments found. Creating new log.
I20260619 06:51:15.172511 30720 log.cc:824] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:51:15.173132 30719 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: No bootstrap required, opened a new log
I20260619 06:51:15.173317 30719 ts_tablet_manager.cc:1397] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 06:51:15.174417 30720 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: No bootstrap required, opened a new log
I20260619 06:51:15.174438 30718 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: No bootstrap required, opened a new log
I20260619 06:51:15.174605 30718 ts_tablet_manager.cc:1397] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 06:51:15.174611 30720 ts_tablet_manager.cc:1397] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:51:15.177091 30719 raft_consensus.cc:348] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:15.177351 30719 raft_consensus.cc:374] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:51:15.177440 30719 raft_consensus.cc:729] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba4fba1f7674725a525b657b93ee784, State: Initialized, Role: FOLLOWER
I20260619 06:51:15.177471 30720 raft_consensus.cc:348] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:15.177752 30720 raft_consensus.cc:374] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:51:15.177811 30720 raft_consensus.cc:729] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0240b0c340354a69b5dc5d08bfd16fe6, State: Initialized, Role: FOLLOWER
I20260619 06:51:15.177874 30719 consensus_queue.cc:260] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:15.178236 30718 raft_consensus.cc:348] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:15.178381 30720 consensus_queue.cc:260] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:15.178530 30718 raft_consensus.cc:374] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:51:15.178639 30718 raft_consensus.cc:729] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7e8db0bfc2464d8694941a0b776da2, State: Initialized, Role: FOLLOWER
I20260619 06:51:15.179018 30719 ts_tablet_manager.cc:1428] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:51:15.179095 30567 heartbeater.cc:503] Master 127.14.116.254:38983 was elected leader, sending a full tablet report...
I20260619 06:51:15.179121 30718 consensus_queue.cc:260] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:15.179294 30720 ts_tablet_manager.cc:1428] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:51:15.179361 30433 heartbeater.cc:503] Master 127.14.116.254:38983 was elected leader, sending a full tablet report...
I20260619 06:51:15.180070 30718 ts_tablet_manager.cc:1428] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:51:15.180431 30701 heartbeater.cc:503] Master 127.14.116.254:38983 was elected leader, sending a full tablet report...
W20260619 06:51:15.190843 30434 tablet.cc:2369] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:51:15.214459 30726 raft_consensus.cc:482] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:51:15.214674 30726 raft_consensus.cc:504] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:15.215763 30726 leader_election.cc:288] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0240b0c340354a69b5dc5d08bfd16fe6 (127.14.116.194:36367), 6ba4fba1f7674725a525b657b93ee784 (127.14.116.193:44701)
I20260619 06:51:15.219381 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.215941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:36367 (local address 127.14.116.195:50601)
0619 06:51:15.216122 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:15.216136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:15.216220 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:15.216520 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:15.216524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:15.216544 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:51:15.216670 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.216680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.217467 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.217470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.218293 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.218302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.218970 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.218974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.219082 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.219120 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:51:15.219177 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 06:51:15.219231 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 06:51:15.219422 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.216037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:50601 (local address 127.14.116.194:36367)
0619 06:51:15.216185 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:15.216190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:15.216206 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:51:15.216347 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:15.216351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:15.216405 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:15.216500 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:51:15.216787 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.217359 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.218426 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.218847 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.218972 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.219025 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:15.219220 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 06:51:15.219303 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:51:15.219504 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.216139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:55969 (local address 127.14.116.193:44701)
0619 06:51:15.216247 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:15.216253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:15.216496 (+ 243us) server_negotiation.cc:407] Connection header received
0619 06:51:15.216598 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:15.216601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:15.216649 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:15.216730 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:51:15.216997 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.217502 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.218497 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.218860 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.218989 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.219037 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:15.219313 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 06:51:15.219382 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:51:15.219645 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.215984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:44701 (local address 127.14.116.195:55969)
0619 06:51:15.216387 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:15.216401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:15.216505 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:15.216766 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:15.216770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:15.216787 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:51:15.216892 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.216898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.217604 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.217606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.218375 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:15.218387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.219003 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.219006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:15.219155 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.219201 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 06:51:15.219277 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:51:15.219341 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":36,"thread_start_us":122,"threads_started":1}
I20260619 06:51:15.220148 30521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5926d463040437fb930f6c544f20449" candidate_uuid: "1c7e8db0bfc2464d8694941a0b776da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" is_pre_election: true
I20260619 06:51:15.220157 30387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5926d463040437fb930f6c544f20449" candidate_uuid: "1c7e8db0bfc2464d8694941a0b776da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba4fba1f7674725a525b657b93ee784" is_pre_election: true
I20260619 06:51:15.220425 30521 raft_consensus.cc:2454] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7e8db0bfc2464d8694941a0b776da2 in term 0.
I20260619 06:51:15.220443 30387 raft_consensus.cc:2454] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7e8db0bfc2464d8694941a0b776da2 in term 0.
I20260619 06:51:15.220856 30590 leader_election.cc:302] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7e8db0bfc2464d8694941a0b776da2, 6ba4fba1f7674725a525b657b93ee784; no voters:
I20260619 06:51:15.221134 30726 raft_consensus.cc:2781] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:51:15.221225 30726 raft_consensus.cc:482] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:51:15.221268 30726 raft_consensus.cc:3037] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:51:15.222240 30726 raft_consensus.cc:504] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:15.222571 30726 leader_election.cc:288] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [CANDIDATE]: Term 1 election: Requested vote from peers 0240b0c340354a69b5dc5d08bfd16fe6 (127.14.116.194:36367), 6ba4fba1f7674725a525b657b93ee784 (127.14.116.193:44701)
I20260619 06:51:15.222903 30521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5926d463040437fb930f6c544f20449" candidate_uuid: "1c7e8db0bfc2464d8694941a0b776da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0240b0c340354a69b5dc5d08bfd16fe6"
I20260619 06:51:15.222944 30387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5926d463040437fb930f6c544f20449" candidate_uuid: "1c7e8db0bfc2464d8694941a0b776da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba4fba1f7674725a525b657b93ee784"
I20260619 06:51:15.223040 30387 raft_consensus.cc:3037] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:51:15.223039 30521 raft_consensus.cc:3037] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:51:15.224323 30387 raft_consensus.cc:2454] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7e8db0bfc2464d8694941a0b776da2 in term 1.
I20260619 06:51:15.224349 30521 raft_consensus.cc:2454] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7e8db0bfc2464d8694941a0b776da2 in term 1.
I20260619 06:51:15.224707 30591 leader_election.cc:302] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0240b0c340354a69b5dc5d08bfd16fe6, 1c7e8db0bfc2464d8694941a0b776da2; no voters:
I20260619 06:51:15.224879 30726 raft_consensus.cc:2781] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:51:15.225076 30726 raft_consensus.cc:686] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c7e8db0bfc2464d8694941a0b776da2, State: Running, Role: LEADER
I20260619 06:51:15.225414 30726 consensus_queue.cc:237] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:15.228478 30241 catalog_manager.cc:5495] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c7e8db0bfc2464d8694941a0b776da2 (127.14.116.195). New cstate: current_term: 1 leader_uuid: "1c7e8db0bfc2464d8694941a0b776da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } health_report { overall_health: UNKNOWN } } }
I20260619 06:51:15.252298 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.248552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.14.116.195:43867)
0619 06:51:15.248688 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:15.248692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:15.248706 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:51:15.248859 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:15.248862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:15.248923 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:15.249028 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:51:15.249280 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.249799 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.250720 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:15.250952 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:15.251060 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:15.251387 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:51:15.251890 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:51:15.252029 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:15.252074 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 06:51:15.252128 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:51:15.258033 30387 raft_consensus.cc:1261] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 1 FOLLOWER]: Refusing update from remote peer 1c7e8db0bfc2464d8694941a0b776da2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:51:15.258035 30521 raft_consensus.cc:1261] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Refusing update from remote peer 1c7e8db0bfc2464d8694941a0b776da2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:51:15.258728 30726 consensus_queue.cc:1035] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:51:15.258916 30728 consensus_queue.cc:1035] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:51:15.266027 30733 mvcc.cc:205] Tried to move back new op lower bound from 7298465281050411008 to 7298465280925782016. Current Snapshot: MvccSnapshot[applied={T|T < 7298465281050411008}]
I20260619 06:51:15.273913 30241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:51:15.274121 30241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:51:15.276536 30241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Sending DeleteTablet for 3 replicas of tablet c5926d463040437fb930f6c544f20449
I20260619 06:51:15.277251 30501 tablet_service.cc:1514] Processing DeleteTablet for tablet c5926d463040437fb930f6c544f20449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:51:15 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:45886
I20260619 06:51:15.277287 30635 tablet_service.cc:1514] Processing DeleteTablet for tablet c5926d463040437fb930f6c544f20449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:51:15 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:46394
I20260619 06:51:15.277379 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 30304
W20260619 06:51:15.283509 30228 connection.cc:535] client connection to 127.14.116.193:44701 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 06:51:15.283655 30228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:51:15.283906 30741 tablet_replica.cc:330] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: stopping tablet replica
I20260619 06:51:15.283972 30740 tablet_replica.cc:330] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: stopping tablet replica
I20260619 06:51:15.284318 30741 raft_consensus.cc:2229] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:51:15.284328 30740 raft_consensus.cc:2229] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:51:15.284354 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 30438
I20260619 06:51:15.284615 30740 raft_consensus.cc:2258] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:51:15.285473 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.285176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:44701 (local address 127.0.0.1:39494)
0619 06:51:15.285303 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:15.285378 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:44701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 06:51:15.285789 30228 catalog_manager.cc:4584] TS 6ba4fba1f7674725a525b657b93ee784 (127.14.116.193:44701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5926d463040437fb930f6c544f20449: Network error: Client connection negotiation failed: client connection to 127.14.116.193:44701: connect: Connection refused (error 111)
I20260619 06:51:15.286242 30741 raft_consensus.cc:2258] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:51:15.287477 30741 ts_tablet_manager.cc:1905] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:51:15.289559 30741 ts_tablet_manager.cc:1918] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:51:15.289669 30741 log.cc:1196] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/c5926d463040437fb930f6c544f20449
I20260619 06:51:15.289978 30741 ts_tablet_manager.cc:1939] T c5926d463040437fb930f6c544f20449 P 1c7e8db0bfc2464d8694941a0b776da2: Deleting consensus metadata
I20260619 06:51:15.290982 30226 catalog_manager.cc:4841] TS 1c7e8db0bfc2464d8694941a0b776da2 (127.14.116.195:43867): tablet c5926d463040437fb930f6c544f20449 (table test-table [id=3e4e1a4f451947e9a0c1b1882bb5d1f8]) successfully deleted
I20260619 06:51:15.291554 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 30572
I20260619 06:51:15.292294 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:15.291785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:36367 (local address 127.0.0.1:45894)
0619 06:51:15.292174 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:15.292224 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.194:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324}
I20260619 06:51:15.297119 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 30210
I20260619 06:51:35.303139 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38983
--webserver_interface=127.14.116.254
--webserver_port=38425
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:38983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:51:35.409996 30743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:35.410326 30743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:35.410406 30743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:35.414386 30743 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:51:35.414461 30743 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:51:35.414489 30743 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:51:35.414520 30743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:51:35.414563 30743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:35.414605 30743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:35.414641 30743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:51:35.414680 30743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:51:35.419819 30743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:38983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=38425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:35.420821 30743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:35.421808 30743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:51:35.434267 30743 server_base.cc:1034] running on GCE node
W20260619 06:51:35.434185 30750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:35.434185 30749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:35.434192 30752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:35.435047 30743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:35.436256 30743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:35.437443 30743 hybrid_clock.cc:648] HybridClock initialized: now 1781851895437406 us; error 54 us; skew 500 ppm
Jun 19 06:51:35 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:35.440209 30743 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:51:35.441246 30743 webserver.cc:460] Webserver started at http://127.14.116.254:38425/ using document root <none> and password file <none>
I20260619 06:51:35.441917 30743 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:35.442013 30743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:35.445878 30743 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:51:35.448032 30759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:35.449256 30743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:51:35.449379 30743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "4bf98da454e54cac98230611a5f4e7bf"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "55bfd8575994fd711d95cf82d362f4c7"
server_key_iv: "1ca7edf99a00a4a2574d12514fdbc3ab"
server_key_version: "encryptionkey@0"
I20260619 06:51:35.449952 30743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:35.458382 30743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:35.461315 30743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:35.461524 30743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:35.468461 30743 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:38983
I20260619 06:51:35.468747 30811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:38983 every 8 connection(s)
I20260619 06:51:35.469614 30743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:51:35.473841 30812 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:51:35.475446 30812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Bootstrap starting.
I20260619 06:51:35.479256 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 30743
I20260619 06:51:35.479830 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:44701
--local_ip_for_outbound_sockets=127.14.116.193
--tserver_master_addrs=127.14.116.254:38983
--webserver_port=37923
--webserver_interface=127.14.116.193
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:51:35.496306 30812 log.cc:824] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:51:35.506736 30812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:51:35.507125 30812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Bootstrap complete.
I20260619 06:51:35.512276 30812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:35.512758 30812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf98da454e54cac98230611a5f4e7bf, State: Initialized, Role: FOLLOWER
I20260619 06:51:35.513278 30812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:35.513412 30812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:51:35.513484 30812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:51:35.513576 30812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:51:35.515880 30812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:35.516216 30812 leader_election.cc:302] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bf98da454e54cac98230611a5f4e7bf; no voters:
I20260619 06:51:35.516525 30812 leader_election.cc:288] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:51:35.516655 30816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:51:35.516925 30816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 2 LEADER]: Becoming Leader. State: Replica: 4bf98da454e54cac98230611a5f4e7bf, State: Running, Role: LEADER
I20260619 06:51:35.517215 30816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:35.517692 30812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:51:35.518055 30818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bf98da454e54cac98230611a5f4e7bf. Latest consensus state: current_term: 2 leader_uuid: "4bf98da454e54cac98230611a5f4e7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } } }
I20260619 06:51:35.518178 30818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:51:35.518456 30817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4bf98da454e54cac98230611a5f4e7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } } }
I20260619 06:51:35.518548 30817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:51:35.519764 30820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:51:35.523411 30820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:51:35.523964 30820 catalog_manager.cc:1259] Loaded cluster ID: 68d8eaecdea44d079a0418f7376052f2
I20260619 06:51:35.524034 30820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:51:35.525759 30820 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:51:35.526439 30820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Loaded TSK: 0
I20260619 06:51:35.537389 30820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Generated new TSK 1
I20260619 06:51:35.537631 30820 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:51:35.608541 30814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:35.608794 30814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:35.608836 30814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:35.612920 30814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:35.612977 30814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:35.613063 30814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:51:35.618518 30814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:44701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=37923
--tserver_master_addrs=127.14.116.254:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:35.619638 30814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:35.620630 30814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:35.630949 30839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:35.631040 30814 server_base.cc:1034] running on GCE node
W20260619 06:51:35.630918 30842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:35.631183 30840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:35.631615 30814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:35.632289 30814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:35.633482 30814 hybrid_clock.cc:648] HybridClock initialized: now 1781851895633457 us; error 42 us; skew 500 ppm
Jun 19 06:51:35 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:35.640633 30814 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:51:35.641672 30814 webserver.cc:460] Webserver started at http://127.14.116.193:37923/ using document root <none> and password file <none>
I20260619 06:51:35.642338 30814 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:35.642450 30814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:35.647318 30814 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:51:35.649883 30849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:35.651322 30814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:51:35.651429 30814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "6ba4fba1f7674725a525b657b93ee784"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "6874efaf4be6069ae19e71b0a56bac41"
server_key_iv: "2646798ff135f41bcaea26237b5241f2"
server_key_version: "encryptionkey@0"
I20260619 06:51:35.652092 30814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:35.676643 30814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:35.680099 30814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:35.680292 30814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:35.680912 30814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:51:35.682111 30856 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 06:51:35.686154 30814 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:51:35.686211 30814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 06:51:35.686316 30814 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:51:35.687917 30814 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:51:35.687960 30814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 06:51:35.688264 30856 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Bootstrap starting.
I20260619 06:51:35.698192 30814 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:44701
I20260619 06:51:35.698223 30963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:44701 every 8 connection(s)
I20260619 06:51:35.699064 30814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:51:35.706807 30856 log.cc:824] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Log is configured to *not* fsync() on all Append() calls
I20260619 06:51:35.707052 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 30814
I20260619 06:51:35.707568 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:36367
--local_ip_for_outbound_sockets=127.14.116.194
--tserver_master_addrs=127.14.116.254:38983
--webserver_port=38823
--webserver_interface=127.14.116.194
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:51:35 dist-test-slave-9x3b krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:51:35.715075 30856 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:51:35.715507 30856 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Bootstrap complete.
I20260619 06:51:35.716332 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:35.701054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:44379 (local address 127.14.116.254:38983)
0619 06:51:35.701393 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:35.701416 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:35.701801 (+ 385us) server_negotiation.cc:407] Connection header received
0619 06:51:35.702858 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:35.702869 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:35.703153 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:35.703481 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:35.704578 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.705550 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.706669 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.707387 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.707555 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:35.710384 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:35.710390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:35.710406 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:35.710439 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:35.712620 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:35.713220 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:35.713226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:35.713231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:35.713379 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:35.713730 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:35.713733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:35.713734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:35.714154 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:35.714301 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:35.714999 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 06:51:35.715392 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":114,"threads_started":1}
I20260619 06:51:35.716361 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:35.701107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:44379)
0619 06:51:35.701636 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:35.701674 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:35.702455 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:35.703531 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:35.703538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:35.703979 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:51:35.704416 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:35.704430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.705714 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.705716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:35.706538 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:35.706547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.707539 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.707546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:35.707789 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:35.708414 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:51:35.708429 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:51:35.710171 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:51:35.712761 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:35.712766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:35.712778 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:35.713092 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:35.713485 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:35.713488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:35.713489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:35.713632 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:35.714296 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:51:35.714305 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:51:35.714642 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 06:51:35.714866 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 06:51:35.715243 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":171,"threads_started":1}
I20260619 06:51:35.716899 30856 ts_tablet_manager.cc:1397] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.012s
I20260619 06:51:35.719097 30964 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38983
I20260619 06:51:35.719317 30964 heartbeater.cc:464] Registering TS with master...
I20260619 06:51:35.719877 30964 heartbeater.cc:511] Master 127.14.116.254:38983 requested a full tablet report, sending...
I20260619 06:51:35.719770 30856 raft_consensus.cc:348] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:35.720239 30856 raft_consensus.cc:729] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba4fba1f7674725a525b657b93ee784, State: Initialized, Role: FOLLOWER
I20260619 06:51:35.720729 30856 consensus_queue.cc:260] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:35.721750 30856 ts_tablet_manager.cc:1428] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:51:35.723675 30776 ts_manager.cc:194] Registered new tserver with Master: 6ba4fba1f7674725a525b657b93ee784 (127.14.116.193:44701)
I20260619 06:51:35.727543 30776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:44379
I20260619 06:51:35.730017 30964 heartbeater.cc:503] Master 127.14.116.254:38983 was elected leader, sending a full tablet report...
Jun 19 06:51:35 dist-test-slave-9x3b krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.193@KRBTEST.COM
I20260619 06:51:35.737340 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:35.726966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:44701 (local address 127.0.0.1:57430)
0619 06:51:35.727406 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:35.727451 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:35.727576 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:35.728232 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:35.728238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:35.728633 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:51:35.728767 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:35.728775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.730218 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.730221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:35.730895 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:35.730905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.731578 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.731582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:35.731707 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:35.732309 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:51:35.732320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:51:35.733630 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:51:35.735666 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:35.735681 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:35.735686 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:35.735930 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:35.736257 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:35.736260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:35.736262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:35.736347 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:35.736860 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:51:35.736865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:51:35.736961 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 06:51:35.737065 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:51:35.737205 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":134,"threads_started":1}
I20260619 06:51:35.737489 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:35.727023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.14.116.193:44701)
0619 06:51:35.727834 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:35.727844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:35.727862 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:51:35.727916 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:35.727920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:35.728067 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:35.728214 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:35.728879 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.730098 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.731022 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.731406 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.731512 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:35.733770 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:35.733779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:35.733791 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:35.733815 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:35.735511 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:35.736040 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:35.736046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:35.736048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:35.736138 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:35.736449 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:35.736452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:35.736454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:35.736752 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:35.736875 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:35.737179 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 06:51:35.737338 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":91,"threads_started":1}
I20260619 06:51:35.738255 30898 tablet_service.cc:1514] Processing DeleteTablet for tablet c5926d463040437fb930f6c544f20449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:51:15 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:57430
I20260619 06:51:35.739054 30973 tablet_replica.cc:330] stopping tablet replica
I20260619 06:51:35.739341 30973 raft_consensus.cc:2229] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:51:35.739590 30973 raft_consensus.cc:2258] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:51:35.740592 30973 ts_tablet_manager.cc:1905] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:51:35.743924 30973 ts_tablet_manager.cc:1918] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:51:35.744031 30973 log.cc:1196] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/c5926d463040437fb930f6c544f20449
I20260619 06:51:35.744295 30973 ts_tablet_manager.cc:1939] T c5926d463040437fb930f6c544f20449 P 6ba4fba1f7674725a525b657b93ee784: Deleting consensus metadata
I20260619 06:51:35.745777 30762 catalog_manager.cc:4841] TS 6ba4fba1f7674725a525b657b93ee784 (127.14.116.193:44701): tablet c5926d463040437fb930f6c544f20449 (table test-table [id=3e4e1a4f451947e9a0c1b1882bb5d1f8]) successfully deleted
W20260619 06:51:35.827232 30969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:35.827466 30969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:35.827507 30969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:35.831791 30969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:35.831841 30969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:35.831925 30969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:51:35.837109 30969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:36367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=38823
--tserver_master_addrs=127.14.116.254:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:35.838131 30969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:35.839103 30969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:35.848344 30979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:35.848472 30969 server_base.cc:1034] running on GCE node
W20260619 06:51:35.848310 30978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:35.848310 30981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:35.848995 30969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:35.849582 30969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:35.850754 30969 hybrid_clock.cc:648] HybridClock initialized: now 1781851895850737 us; error 26 us; skew 500 ppm
Jun 19 06:51:35 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:35.853510 30969 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:51:35.854476 30969 webserver.cc:460] Webserver started at http://127.14.116.194:38823/ using document root <none> and password file <none>
I20260619 06:51:35.855031 30969 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:35.855123 30969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:35.858978 30969 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:51:35.861056 30988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:35.862282 30969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:51:35.862385 30969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "0240b0c340354a69b5dc5d08bfd16fe6"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "fc6a4ec3ae5da2948a03816d36d2e1cb"
server_key_iv: "e0550648603b72796e7eaab503f040d3"
server_key_version: "encryptionkey@0"
I20260619 06:51:35.862924 30969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:35.883345 30969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:35.886266 30969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:35.886462 30969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:35.887038 30969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:51:35.888130 30995 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 06:51:35.891963 30969 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 06:51:35.892017 30969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 06:51:35.892086 30969 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 06:51:35.893638 30969 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 06:51:35.893688 30969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 06:51:35.893940 30995 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Bootstrap starting.
I20260619 06:51:35.904163 30969 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:36367
I20260619 06:51:35.904228 31102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:36367 every 8 connection(s)
I20260619 06:51:35.905140 30969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:51:35.912757 30995 log.cc:824] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:51:35.914911 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 30969
Jun 19 06:51:35 dist-test-slave-9x3b krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:51:35.915428 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:43867
--local_ip_for_outbound_sockets=127.14.116.195
--tserver_master_addrs=127.14.116.254:38983
--webserver_port=33727
--webserver_interface=127.14.116.195
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:51:35.920663 30995 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:51:35.921060 30995 tablet_bootstrap.cc:492] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Bootstrap complete.
I20260619 06:51:35.921093 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:35.907216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:60327)
0619 06:51:35.907806 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:35.907868 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:35.908925 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:35.909500 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:35.909509 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:35.909982 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:51:35.910735 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:35.910756 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.911584 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.911587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:35.912912 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:35.912922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.913420 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.913424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:35.913613 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:35.914253 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:51:35.914274 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:51:35.915882 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:51:35.918026 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:35.918031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:35.918044 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:35.918385 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:35.918756 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:35.918759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:35.918761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:35.918940 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:35.919427 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:51:35.919433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:51:35.919709 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 06:51:35.919883 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 06:51:35.920167 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":101,"threads_started":1}
I20260619 06:51:35.922046 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:35.907083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:60327 (local address 127.14.116.254:38983)
0619 06:51:35.907235 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:35.907240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:35.907982 (+ 742us) server_negotiation.cc:407] Connection header received
0619 06:51:35.909174 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:35.909177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:35.909227 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:35.909309 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:35.910906 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.911438 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.913036 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.913278 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.913370 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:35.916047 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:35.916051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:35.916053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:35.916087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:35.917901 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:35.918511 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:35.918518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:35.918522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:35.918615 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:35.919054 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:35.919060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:35.919065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:35.919311 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:35.919413 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:35.921647 (+ 2234us) server_negotiation.cc:299] Negotiation successful
0619 06:51:35.921801 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:51:35.922441 30995 ts_tablet_manager.cc:1397] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.008s
I20260619 06:51:35.923044 31103 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38983
I20260619 06:51:35.923265 31103 heartbeater.cc:464] Registering TS with master...
I20260619 06:51:35.923784 31103 heartbeater.cc:511] Master 127.14.116.254:38983 requested a full tablet report, sending...
I20260619 06:51:35.925118 30995 raft_consensus.cc:348] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:35.925590 30995 raft_consensus.cc:729] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0240b0c340354a69b5dc5d08bfd16fe6, State: Initialized, Role: FOLLOWER
I20260619 06:51:35.925607 30776 ts_manager.cc:194] Registered new tserver with Master: 0240b0c340354a69b5dc5d08bfd16fe6 (127.14.116.194:36367)
I20260619 06:51:35.926018 30995 consensus_queue.cc:260] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } }
I20260619 06:51:35.926873 30995 ts_tablet_manager.cc:1428] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 06:51:35.927069 30776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:60327
I20260619 06:51:35.929263 31103 heartbeater.cc:503] Master 127.14.116.254:38983 was elected leader, sending a full tablet report...
I20260619 06:51:35.931838 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:35.927305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:36367 (local address 127.0.0.1:39398)
0619 06:51:35.927478 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:35.927493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:35.927567 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:35.928514 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:35.928517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:35.928536 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:51:35.928623 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:35.928629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.929819 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.929825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:35.930604 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:35.930613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.931361 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.931365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:35.931459 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:35.931517 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 06:51:35.931619 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:51:35.931719 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 06:51:35.931859 31109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:35.927383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.14.116.194:36367)
0619 06:51:35.927810 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:35.927821 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:35.927840 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:51:35.927900 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:35.927905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:35.928350 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:35.928500 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:51:35.928743 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.929722 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.930733 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:35.931234 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:35.931372 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:35.931485 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:35.931663 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 06:51:35.931740 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":96,"threads_started":1}
I20260619 06:51:35.932713 31037 tablet_service.cc:1514] Processing DeleteTablet for tablet c5926d463040437fb930f6c544f20449 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:51:15 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:39398
I20260619 06:51:35.933470 31110 tablet_replica.cc:330] stopping tablet replica
I20260619 06:51:35.933754 31110 raft_consensus.cc:2229] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:51:35.933971 31110 raft_consensus.cc:2258] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:51:35.934938 31110 ts_tablet_manager.cc:1905] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:51:35.939214 31110 ts_tablet_manager.cc:1918] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:51:35.939365 31110 log.cc:1196] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/c5926d463040437fb930f6c544f20449
I20260619 06:51:35.939656 31110 ts_tablet_manager.cc:1939] T c5926d463040437fb930f6c544f20449 P 0240b0c340354a69b5dc5d08bfd16fe6: Deleting consensus metadata
I20260619 06:51:35.941376 30763 catalog_manager.cc:4841] TS 0240b0c340354a69b5dc5d08bfd16fe6 (127.14.116.194:36367): tablet c5926d463040437fb930f6c544f20449 (table test-table [id=3e4e1a4f451947e9a0c1b1882bb5d1f8]) successfully deleted
W20260619 06:51:36.030130 31107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:36.030364 31107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:36.030438 31107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:36.034509 31107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:36.034565 31107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:36.034678 31107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:51:36.040089 31107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:43867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=33727
--tserver_master_addrs=127.14.116.254:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:36.041036 31107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:36.042004 31107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:36.051133 31115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:36.051127 31116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:36.051239 31107 server_base.cc:1034] running on GCE node
W20260619 06:51:36.051127 31118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:36.051748 31107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:36.052359 31107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:36.053509 31107 hybrid_clock.cc:648] HybridClock initialized: now 1781851896053498 us; error 22 us; skew 500 ppm
Jun 19 06:51:36 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851896, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:36.056154 31107 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:51:36.057018 31107 webserver.cc:460] Webserver started at http://127.14.116.195:33727/ using document root <none> and password file <none>
I20260619 06:51:36.057622 31107 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:36.057694 31107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:36.061309 31107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:51:36.063117 31125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:36.064280 31107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:51:36.064414 31107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "1c7e8db0bfc2464d8694941a0b776da2"
format_stamp: "Formatted at 2026-06-19 06:51:15 on dist-test-slave-9x3b"
server_key: "3ecef243ed74ce300e236ad0a7372b7e"
server_key_iv: "a7493993a14a0ea48a21b96232557d1d"
server_key_version: "encryptionkey@0"
I20260619 06:51:36.064908 31107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:36.075132 31107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:36.077651 31107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:36.077833 31107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:36.078346 31107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:51:36.079077 31107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:51:36.079154 31107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:36.079241 31107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:51:36.079291 31107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:36.087780 31107 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:43867
I20260619 06:51:36.087834 31238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:43867 every 8 connection(s)
I20260619 06:51:36.088670 31107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:51:36.092868 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 31107
Jun 19 06:51:36 dist-test-slave-9x3b krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851896, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:51:36.101630 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.093532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.14.116.254:38983)
0619 06:51:36.093739 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:36.093745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:36.093925 (+ 180us) server_negotiation.cc:407] Connection header received
0619 06:51:36.094091 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:36.094095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:36.094139 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:36.094245 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:36.094950 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.095468 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.096414 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.096734 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.096812 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.098227 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:36.098231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:36.098233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:36.098259 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:36.099901 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:36.100361 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:36.100363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:36.100364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:36.100474 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:36.100760 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:36.100763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:36.100766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:36.101000 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:36.101175 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:36.101368 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 06:51:36.101503 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"spinlock_wait_cycles":5760,"thread_start_us":73,"threads_started":1}
I20260619 06:51:36.102649 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.090274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:43601 (local address 127.14.116.254:38983)
0619 06:51:36.090416 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:36.090420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:36.091568 (+ 1148us) server_negotiation.cc:407] Connection header received
0619 06:51:36.092453 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:36.092456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:36.092514 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:36.092620 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:36.093972 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.094500 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.095572 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.095844 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.095953 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.098606 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:36.098610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:36.098614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:36.098653 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:36.100412 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:36.100962 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:36.100964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:36.100966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:36.101035 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:36.101453 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:36.101457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:36.101459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:36.101652 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:36.101770 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:36.102411 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 06:51:36.102521 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":3840}
I20260619 06:51:36.103566 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.090614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:43601)
0619 06:51:36.091433 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:36.091471 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:36.092258 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:36.092788 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:36.092802 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:36.093187 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:51:36.093832 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:36.093848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.094622 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.094625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:36.095452 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:36.095460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.095985 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.095988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:36.096145 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.096671 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:51:36.096685 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:51:36.098421 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:51:36.100507 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:36.100516 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:36.100533 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:36.100857 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:36.101156 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:51:36.101159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:51:36.101161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:51:36.101346 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:51:36.101936 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:51:36.101941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:51:36.102150 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 06:51:36.102360 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 06:51:36.102644 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":515,"threads_started":1}
I20260619 06:51:36.104590 31239 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38983
I20260619 06:51:36.104791 31239 heartbeater.cc:464] Registering TS with master...
I20260619 06:51:36.105190 31239 heartbeater.cc:511] Master 127.14.116.254:38983 requested a full tablet report, sending...
I20260619 06:51:36.106094 30776 ts_manager.cc:194] Registered new tserver with Master: 1c7e8db0bfc2464d8694941a0b776da2 (127.14.116.195:43867)
I20260619 06:51:36.106587 30776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:43601
I20260619 06:51:36.115105 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:51:36.118808 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.115668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.14.116.254:38983)
0619 06:51:36.115780 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:36.115784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:36.115988 (+ 204us) server_negotiation.cc:407] Connection header received
0619 06:51:36.116114 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:36.116116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:36.116204 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:36.116280 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:51:36.116490 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.116959 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.117746 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.117944 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.118013 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.118371 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:51:36.118534 (+ 163us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 06:51:36.118721 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42620: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 06:51:36.118880 30966 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42620: token expired
W20260619 06:51:36.119211 30705 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.116.254:38983)
I20260619 06:51:36.126288 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.119580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.14.116.254:38983)
0619 06:51:36.119697 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:36.119702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:36.119744 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:51:36.119869 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:36.119872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:36.119910 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:36.119995 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:51:36.120585 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.121056 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.121832 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.122054 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.122157 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.123395 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:51:36.123398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:51:36.123401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:51:36.123424 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:51:36.124722 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:36.125143 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:36.125147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:36.125149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:36.125227 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:51:36.125494 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:51:36.125498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:51:36.125500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:51:36.125667 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:51:36.125795 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:36.126026 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 06:51:36.126131 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:51:36.128563 30776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:51:36.131845 30776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:51:36.138775 30898 tablet_service.cc:1467] Processing CreateTablet for tablet 1daeb797ce3f45c19a2f9b7aab6b762b (DEFAULT_TABLE table=test-table [id=0adb6f47d28e4e66bdf8b7d7d3da1696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:51:36.138839 31037 tablet_service.cc:1467] Processing CreateTablet for tablet 1daeb797ce3f45c19a2f9b7aab6b762b (DEFAULT_TABLE table=test-table [id=0adb6f47d28e4e66bdf8b7d7d3da1696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:51:36.139372 30898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1daeb797ce3f45c19a2f9b7aab6b762b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:51:36.139371 31037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1daeb797ce3f45c19a2f9b7aab6b762b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:51:36.142089 30856 tablet_bootstrap.cc:492] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: Bootstrap starting.
I20260619 06:51:36.142517 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.138087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:43867 (local address 127.0.0.1:42062)
0619 06:51:36.138236 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:36.138250 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:36.138317 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:36.139060 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:36.139067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:36.139084 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:51:36.139163 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:36.139167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.140610 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.140616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:36.141303 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:36.141310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.142173 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.142176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:36.142260 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.142282 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:51:36.142333 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 06:51:36.142401 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 06:51:36.142587 30995 tablet_bootstrap.cc:492] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: Bootstrap starting.
I20260619 06:51:36.142989 30856 tablet_bootstrap.cc:654] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: Neither blocks nor log segments found. Creating new log.
I20260619 06:51:36.143220 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.138301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.14.116.195:43867)
0619 06:51:36.138630 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:36.138641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:36.138663 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:51:36.138732 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:36.138736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:36.138900 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:36.139283 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:51:36.139434 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.140505 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.141437 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.142016 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.142770 (+ 754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.142913 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:36.142968 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 06:51:36.143059 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":158,"threads_started":1}
I20260619 06:51:36.143373 30995 tablet_bootstrap.cc:654] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: Neither blocks nor log segments found. Creating new log.
I20260619 06:51:36.144853 30995 tablet_bootstrap.cc:492] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: No bootstrap required, opened a new log
I20260619 06:51:36.144919 30995 ts_tablet_manager.cc:1397] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:51:36.145315 30995 raft_consensus.cc:348] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } }
I20260619 06:51:36.145409 30995 raft_consensus.cc:374] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:51:36.145435 30995 raft_consensus.cc:729] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0240b0c340354a69b5dc5d08bfd16fe6, State: Initialized, Role: FOLLOWER
I20260619 06:51:36.145597 30995 consensus_queue.cc:260] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } }
I20260619 06:51:36.145854 30995 ts_tablet_manager.cc:1428] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:51:36.146246 31173 tablet_service.cc:1467] Processing CreateTablet for tablet 1daeb797ce3f45c19a2f9b7aab6b762b (DEFAULT_TABLE table=test-table [id=0adb6f47d28e4e66bdf8b7d7d3da1696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:51:36.147248 31173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1daeb797ce3f45c19a2f9b7aab6b762b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:51:36.148152 30856 tablet_bootstrap.cc:492] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: No bootstrap required, opened a new log
I20260619 06:51:36.148233 30856 ts_tablet_manager.cc:1397] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 06:51:36.148634 30856 raft_consensus.cc:348] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } }
I20260619 06:51:36.148731 30856 raft_consensus.cc:374] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:51:36.148763 30856 raft_consensus.cc:729] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba4fba1f7674725a525b657b93ee784, State: Initialized, Role: FOLLOWER
I20260619 06:51:36.148881 30856 consensus_queue.cc:260] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } }
I20260619 06:51:36.149111 30856 ts_tablet_manager.cc:1428] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:51:36.153906 31251 tablet_bootstrap.cc:492] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: Bootstrap starting.
W20260619 06:51:36.156026 31104 tablet.cc:2369] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:51:36.156248 31251 tablet_bootstrap.cc:654] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: Neither blocks nor log segments found. Creating new log.
I20260619 06:51:36.157090 31251 log.cc:824] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:51:36.158679 31251 tablet_bootstrap.cc:492] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: No bootstrap required, opened a new log
I20260619 06:51:36.158823 31251 ts_tablet_manager.cc:1397] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 06:51:36.161443 31251 raft_consensus.cc:348] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } }
I20260619 06:51:36.161674 31251 raft_consensus.cc:374] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:51:36.161739 31251 raft_consensus.cc:729] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7e8db0bfc2464d8694941a0b776da2, State: Initialized, Role: FOLLOWER
I20260619 06:51:36.162150 31251 consensus_queue.cc:260] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } }
I20260619 06:51:36.162709 31239 heartbeater.cc:503] Master 127.14.116.254:38983 was elected leader, sending a full tablet report...
I20260619 06:51:36.162943 31251 ts_tablet_manager.cc:1428] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260619 06:51:36.164461 30970 raft_consensus.cc:482] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:51:36.164618 30970 raft_consensus.cc:504] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } }
I20260619 06:51:36.165792 30970 leader_election.cc:288] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c7e8db0bfc2464d8694941a0b776da2 (127.14.116.195:43867), 0240b0c340354a69b5dc5d08bfd16fe6 (127.14.116.194:36367)
I20260619 06:51:36.170125 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.166150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:53473 (local address 127.14.116.195:43867)
0619 06:51:36.166250 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:36.166254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:36.166269 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:51:36.166400 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:36.166402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:36.166433 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:36.166491 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:51:36.166743 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.167471 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.169041 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.169558 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.169679 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.169723 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:36.169948 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 06:51:36.170006 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:51:36.170153 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.165929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:43867 (local address 127.14.116.193:53473)
0619 06:51:36.166172 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:36.166186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:36.166281 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:36.166502 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:36.166505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:36.166521 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:51:36.166633 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:36.166642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.167606 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.167612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:36.168905 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:36.168915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.169673 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.169676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:36.169804 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.169848 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 06:51:36.169935 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:51:36.170007 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260619 06:51:36.170174 31109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.166470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:58919 (local address 127.14.116.194:36367)
0619 06:51:36.166607 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:36.166611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:36.166623 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:51:36.166667 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:36.166669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:36.166726 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:36.166803 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:51:36.167072 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.167786 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.169147 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.169617 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.169742 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.169789 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:36.169999 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 06:51:36.170065 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:51:36.170197 31254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.165993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:36367 (local address 127.14.116.193:58919)
0619 06:51:36.166300 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:36.166312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:36.166388 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:36.166826 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:51:36.166829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:51:36.166846 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:51:36.166960 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:36.166967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.167873 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.167876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:36.169027 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:51:36.169034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.169745 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.169749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:51:36.169883 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.169901 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 06:51:36.169987 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:51:36.170053 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260619 06:51:36.170745 31057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1daeb797ce3f45c19a2f9b7aab6b762b" candidate_uuid: "6ba4fba1f7674725a525b657b93ee784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" is_pre_election: true
I20260619 06:51:36.170995 31057 raft_consensus.cc:2454] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ba4fba1f7674725a525b657b93ee784 in term 0.
I20260619 06:51:36.171041 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1daeb797ce3f45c19a2f9b7aab6b762b" candidate_uuid: "6ba4fba1f7674725a525b657b93ee784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7e8db0bfc2464d8694941a0b776da2" is_pre_election: true
I20260619 06:51:36.171276 31193 raft_consensus.cc:2454] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ba4fba1f7674725a525b657b93ee784 in term 0.
I20260619 06:51:36.171376 30853 leader_election.cc:302] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0240b0c340354a69b5dc5d08bfd16fe6, 6ba4fba1f7674725a525b657b93ee784; no voters:
I20260619 06:51:36.171638 30970 raft_consensus.cc:2781] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:51:36.171736 30970 raft_consensus.cc:482] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:51:36.171815 30970 raft_consensus.cc:3037] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:51:36.172715 30970 raft_consensus.cc:504] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } }
I20260619 06:51:36.173046 30970 leader_election.cc:288] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [CANDIDATE]: Term 1 election: Requested vote from peers 1c7e8db0bfc2464d8694941a0b776da2 (127.14.116.195:43867), 0240b0c340354a69b5dc5d08bfd16fe6 (127.14.116.194:36367)
I20260619 06:51:36.173365 31193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1daeb797ce3f45c19a2f9b7aab6b762b" candidate_uuid: "6ba4fba1f7674725a525b657b93ee784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7e8db0bfc2464d8694941a0b776da2"
I20260619 06:51:36.173404 31057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1daeb797ce3f45c19a2f9b7aab6b762b" candidate_uuid: "6ba4fba1f7674725a525b657b93ee784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0240b0c340354a69b5dc5d08bfd16fe6"
I20260619 06:51:36.173452 31193 raft_consensus.cc:3037] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:51:36.173545 31057 raft_consensus.cc:3037] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:51:36.174386 31193 raft_consensus.cc:2454] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ba4fba1f7674725a525b657b93ee784 in term 1.
I20260619 06:51:36.174563 31057 raft_consensus.cc:2454] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ba4fba1f7674725a525b657b93ee784 in term 1.
I20260619 06:51:36.174676 30850 leader_election.cc:302] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7e8db0bfc2464d8694941a0b776da2, 6ba4fba1f7674725a525b657b93ee784; no voters:
I20260619 06:51:36.174875 30970 raft_consensus.cc:2781] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:51:36.175046 30970 raft_consensus.cc:686] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 1 LEADER]: Becoming Leader. State: Replica: 6ba4fba1f7674725a525b657b93ee784, State: Running, Role: LEADER
I20260619 06:51:36.175269 30970 consensus_queue.cc:237] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } }
I20260619 06:51:36.177759 30775 catalog_manager.cc:5495] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ba4fba1f7674725a525b657b93ee784 (127.14.116.193). New cstate: current_term: 1 leader_uuid: "6ba4fba1f7674725a525b657b93ee784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ba4fba1f7674725a525b657b93ee784" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 } health_report { overall_health: UNKNOWN } } }
I20260619 06:51:36.194607 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:36.191538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.14.116.193:44701)
0619 06:51:36.191676 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 06:51:36.191679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:51:36.191691 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:51:36.191791 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:51:36.191794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:51:36.191841 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:51:36.191933 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:51:36.192121 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.192624 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.193379 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:51:36.193591 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:51:36.193662 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:51:36.193864 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:51:36.194252 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:51:36.194331 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:51:36.194398 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 06:51:36.194442 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:51:36.198199 31057 raft_consensus.cc:1261] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Refusing update from remote peer 6ba4fba1f7674725a525b657b93ee784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:51:36.198202 31193 raft_consensus.cc:1261] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [term 1 FOLLOWER]: Refusing update from remote peer 6ba4fba1f7674725a525b657b93ee784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:51:36.198843 30970 consensus_queue.cc:1035] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0240b0c340354a69b5dc5d08bfd16fe6" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:51:36.199003 31255 consensus_queue.cc:1035] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7e8db0bfc2464d8694941a0b776da2" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 43867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:51:36.200213 30965 tablet.cc:2369] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:51:36.204558 31258 mvcc.cc:205] Tried to move back new op lower bound from 7298465366821748736 to 7298465366736367616. Current Snapshot: MvccSnapshot[applied={T|T < 7298465366821748736}]
I20260619 06:51:36.213759 30774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:51:36.213939 30774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:51:36.216248 30774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Sending DeleteTablet for 3 replicas of tablet 1daeb797ce3f45c19a2f9b7aab6b762b
I20260619 06:51:36.216861 30898 tablet_service.cc:1514] Processing DeleteTablet for tablet 1daeb797ce3f45c19a2f9b7aab6b762b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:51:36 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:57430
I20260619 06:51:36.216871 31173 tablet_service.cc:1514] Processing DeleteTablet for tablet 1daeb797ce3f45c19a2f9b7aab6b762b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:51:36 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:42062
I20260619 06:51:36.216979 31037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1daeb797ce3f45c19a2f9b7aab6b762b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:51:36 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:39398
I20260619 06:51:36.217028 30973 tablet_replica.cc:330] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: stopping tablet replica
I20260619 06:51:36.217129 31110 tablet_replica.cc:330] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: stopping tablet replica
I20260619 06:51:36.217129 30973 raft_consensus.cc:2229] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:51:36.217212 31110 raft_consensus.cc:2229] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:51:36.217371 31110 raft_consensus.cc:2258] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:51:36.217378 30973 raft_consensus.cc:2258] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:51:36.218204 31110 ts_tablet_manager.cc:1905] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:51:36.218413 30973 ts_tablet_manager.cc:1905] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:51:36.220451 30973 ts_tablet_manager.cc:1918] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:51:36.220541 30973 log.cc:1196] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal/wals/1daeb797ce3f45c19a2f9b7aab6b762b
I20260619 06:51:36.220706 31110 ts_tablet_manager.cc:1918] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:51:36.220788 31110 log.cc:1196] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/1daeb797ce3f45c19a2f9b7aab6b762b
I20260619 06:51:36.220804 30973 ts_tablet_manager.cc:1939] T 1daeb797ce3f45c19a2f9b7aab6b762b P 6ba4fba1f7674725a525b657b93ee784: Deleting consensus metadata
I20260619 06:51:36.221056 31110 ts_tablet_manager.cc:1939] T 1daeb797ce3f45c19a2f9b7aab6b762b P 0240b0c340354a69b5dc5d08bfd16fe6: Deleting consensus metadata
I20260619 06:51:36.221472 30762 catalog_manager.cc:4841] TS 6ba4fba1f7674725a525b657b93ee784 (127.14.116.193:44701): tablet 1daeb797ce3f45c19a2f9b7aab6b762b (table test-table [id=0adb6f47d28e4e66bdf8b7d7d3da1696]) successfully deleted
I20260619 06:51:36.221499 31268 tablet_replica.cc:330] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: stopping tablet replica
I20260619 06:51:36.221668 30763 catalog_manager.cc:4841] TS 0240b0c340354a69b5dc5d08bfd16fe6 (127.14.116.194:36367): tablet 1daeb797ce3f45c19a2f9b7aab6b762b (table test-table [id=0adb6f47d28e4e66bdf8b7d7d3da1696]) successfully deleted
I20260619 06:51:36.221817 31268 raft_consensus.cc:2229] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:51:36.221975 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 30814
I20260619 06:51:36.222141 31268 raft_consensus.cc:2258] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:51:36.224308 31268 ts_tablet_manager.cc:1905] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:51:36.226229 31268 ts_tablet_manager.cc:1918] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:51:36.226334 31268 log.cc:1196] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/1daeb797ce3f45c19a2f9b7aab6b762b
I20260619 06:51:36.226627 31268 ts_tablet_manager.cc:1939] T 1daeb797ce3f45c19a2f9b7aab6b762b P 1c7e8db0bfc2464d8694941a0b776da2: Deleting consensus metadata
I20260619 06:51:36.227578 30760 catalog_manager.cc:4841] TS 1c7e8db0bfc2464d8694941a0b776da2 (127.14.116.195:43867): tablet 1daeb797ce3f45c19a2f9b7aab6b762b (table test-table [id=0adb6f47d28e4e66bdf8b7d7d3da1696]) successfully deleted
I20260619 06:51:36.228178 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 30969
I20260619 06:51:36.233624 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 31107
I20260619 06:51:36.238946 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 30743
I20260619 06:51:56.245298 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38983
--webserver_interface=127.14.116.254
--webserver_port=38425
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:38983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:51:56.356411 31269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:56.356701 31269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:56.356786 31269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:56.360783 31269 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:51:56.360854 31269 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:51:56.360878 31269 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:51:56.360900 31269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:51:56.360944 31269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:56.360988 31269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:56.361027 31269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:51:56.361063 31269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:51:56.366192 31269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:38983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=38425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:56.367208 31269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:56.368266 31269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:56.376490 31274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:56.376533 31275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:56.376549 31277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:56.376766 31269 server_base.cc:1034] running on GCE node
I20260619 06:51:56.377430 31269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:56.378464 31269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:56.379657 31269 hybrid_clock.cc:648] HybridClock initialized: now 1781851916379638 us; error 27 us; skew 500 ppm
Jun 19 06:51:56 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851916, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:56.387434 31269 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:51:56.388397 31269 webserver.cc:460] Webserver started at http://127.14.116.254:38425/ using document root <none> and password file <none>
I20260619 06:51:56.388974 31269 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:56.389070 31269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:56.393072 31269 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:51:56.395188 31284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:56.396397 31269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:51:56.396544 31269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "4bf98da454e54cac98230611a5f4e7bf"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "55bfd8575994fd711d95cf82d362f4c7"
server_key_iv: "1ca7edf99a00a4a2574d12514fdbc3ab"
server_key_version: "encryptionkey@0"
I20260619 06:51:56.397092 31269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:56.410375 31269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:56.413515 31269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:56.413712 31269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:56.420399 31269 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:38983
I20260619 06:51:56.420687 31336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:38983 every 8 connection(s)
I20260619 06:51:56.421461 31269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:51:56.425562 31337 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:51:56.427049 31337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Bootstrap starting.
I20260619 06:51:56.431509 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 31269
I20260619 06:51:56.431967 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:44701
--local_ip_for_outbound_sockets=127.14.116.193
--tserver_master_addrs=127.14.116.254:38983
--webserver_port=37923
--webserver_interface=127.14.116.193
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:51:56.444665 31337 log.cc:824] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:51:56.455315 31337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:51:56.455781 31337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Bootstrap complete.
I20260619 06:51:56.459708 31337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:56.460079 31337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf98da454e54cac98230611a5f4e7bf, State: Initialized, Role: FOLLOWER
I20260619 06:51:56.460600 31337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:56.460716 31337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:51:56.460784 31337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:51:56.460879 31337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:51:56.462652 31337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:56.462998 31337 leader_election.cc:302] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bf98da454e54cac98230611a5f4e7bf; no voters:
I20260619 06:51:56.463284 31337 leader_election.cc:288] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:51:56.463438 31341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:51:56.463696 31341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [term 3 LEADER]: Becoming Leader. State: Replica: 4bf98da454e54cac98230611a5f4e7bf, State: Running, Role: LEADER
I20260619 06:51:56.464000 31341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } }
I20260619 06:51:56.464465 31337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:51:56.465288 31342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4bf98da454e54cac98230611a5f4e7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } } }
I20260619 06:51:56.465411 31342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:51:56.466090 31343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bf98da454e54cac98230611a5f4e7bf. Latest consensus state: current_term: 3 leader_uuid: "4bf98da454e54cac98230611a5f4e7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf98da454e54cac98230611a5f4e7bf" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38983 } } }
I20260619 06:51:56.466181 31343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:51:56.466697 31350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:51:56.471750 31350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:51:56.472343 31350 catalog_manager.cc:1259] Loaded cluster ID: 68d8eaecdea44d079a0418f7376052f2
I20260619 06:51:56.472400 31350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:51:56.475327 31350 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:51:56.476056 31350 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Loaded TSK: 0
I20260619 06:51:56.476150 31350 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4bf98da454e54cac98230611a5f4e7bf: Loaded TSK: 1
F20260619 06:51:56.477049 31350 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4e1b1179ad google::LogMessage::Fail() at ??:0
@ 0x7f4e1b11a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4e1b1174de google::LogMessage::Flush() at ??:0
@ 0x7f4e1b118a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4e1dddb18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4e1ddd3f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4e25a7c7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4e25a5699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4e25a57178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f4e25a98df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4e25ab782b std::function<>::operator()() at ??:0
@ 0x7f4e25a56b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4e25a57f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4e25a54579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4e25a98640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4e1d3693d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4e1d369c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4e1d36b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4e1d35a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4e1e99e6db start_thread at ??:0
@ 0x7f4e19c1561f clone at ??:0
W20260619 06:51:56.579028 31339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:56.579267 31339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:56.579336 31339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:56.583343 31339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:56.583392 31339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:56.583473 31339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:51:56.588599 31339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:44701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=37923
--tserver_master_addrs=127.14.116.254:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:56.589689 31339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:56.590696 31339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:56.601019 31386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:56.601125 31388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:56.601267 31339 server_base.cc:1034] running on GCE node
W20260619 06:51:56.601063 31385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:56.601923 31339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:56.602661 31339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:56.603850 31339 hybrid_clock.cc:648] HybridClock initialized: now 1781851916603815 us; error 43 us; skew 500 ppm
Jun 19 06:51:56 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851916, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:56.606997 31339 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:51:56.607890 31339 webserver.cc:460] Webserver started at http://127.14.116.193:37923/ using document root <none> and password file <none>
I20260619 06:51:56.608518 31339 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:56.608580 31339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:56.612289 31339 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:51:56.614336 31395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:56.615573 31339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:51:56.615728 31339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "6ba4fba1f7674725a525b657b93ee784"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "6874efaf4be6069ae19e71b0a56bac41"
server_key_iv: "2646798ff135f41bcaea26237b5241f2"
server_key_version: "encryptionkey@0"
I20260619 06:51:56.616240 31339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:56.633750 31339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:56.636929 31339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:56.637112 31339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:56.637657 31339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:51:56.638392 31339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:51:56.638459 31339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:56.638522 31339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:51:56.638573 31339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:56.648669 31339 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:44701
I20260619 06:51:56.648741 31508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:44701 every 8 connection(s)
I20260619 06:51:56.649621 31339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:51:56.659082 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 31339
I20260619 06:51:56.659540 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:36367
--local_ip_for_outbound_sockets=127.14.116.194
--tserver_master_addrs=127.14.116.254:38983
--webserver_port=38823
--webserver_interface=127.14.116.194
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:51:56.691252 31511 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:51:56.651675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:41185)
0619 06:51:56.652421 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:56.652487 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 06:51:56.653552 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:51:56.690503 (+ 36951us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":111,"threads_started":1}
W20260619 06:51:56.691824 31509 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:38983 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.254:38983: Client connection negotiation failed: client connection to 127.14.116.254:38983: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:51:56.692693 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:56.692315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:32953)
0619 06:51:56.692526 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:56.692612 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 06:51:56.693501 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:56.693290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:50573)
0619 06:51:56.693412 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:56.693454 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:51:56.693672 31509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:51:56.769414 31512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:56.769701 31512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:56.769778 31512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:56.773799 31512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:56.773989 31512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:56.774148 31512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:51:56.779603 31512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:36367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=38823
--tserver_master_addrs=127.14.116.254:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:56.780678 31512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:56.781697 31512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:56.790933 31518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:56.790954 31520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:56.790970 31517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:56.791198 31512 server_base.cc:1034] running on GCE node
I20260619 06:51:56.791679 31512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:56.792243 31512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:56.793437 31512 hybrid_clock.cc:648] HybridClock initialized: now 1781851916793426 us; error 29 us; skew 500 ppm
Jun 19 06:51:56 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851916, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:56.796018 31512 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:51:56.796938 31512 webserver.cc:460] Webserver started at http://127.14.116.194:38823/ using document root <none> and password file <none>
I20260619 06:51:56.797425 31512 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:56.797480 31512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:56.800940 31512 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:51:56.802814 31527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:56.804014 31512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:51:56.804128 31512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "0240b0c340354a69b5dc5d08bfd16fe6"
format_stamp: "Formatted at 2026-06-19 06:51:14 on dist-test-slave-9x3b"
server_key: "fc6a4ec3ae5da2948a03816d36d2e1cb"
server_key_iv: "e0550648603b72796e7eaab503f040d3"
server_key_version: "encryptionkey@0"
I20260619 06:51:56.804657 31512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:56.853153 31512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:56.855994 31512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:56.856190 31512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:56.856786 31512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:51:56.857632 31512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:51:56.857677 31512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:56.857744 31512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:51:56.857775 31512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:56.866514 31512 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:36367
I20260619 06:51:56.866549 31640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:36367 every 8 connection(s)
I20260619 06:51:56.867478 31512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:51:56.871840 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:56.869777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:46743)
0619 06:51:56.870723 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:56.870890 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":224,"threads_started":1}
W20260619 06:51:56.872246 31641 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:38983 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.254:38983: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
I20260619 06:51:56.872799 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:56.872583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:47695)
0619 06:51:56.872693 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:56.872739 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 06:51:56.873370 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:56.873172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:36923)
0619 06:51:56.873265 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:56.873312 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:51:56.873588 31641 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:51:56.876237 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 31512
I20260619 06:51:56.876662 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:43867
--local_ip_for_outbound_sockets=127.14.116.195
--tserver_master_addrs=127.14.116.254:38983
--webserver_port=33727
--webserver_interface=127.14.116.195
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:51:56.982406 31644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:51:56.982689 31644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:51:56.982857 31644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:51:56.986945 31644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:51:56.987035 31644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:51:56.987124 31644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:51:56.992162 31644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:43867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=33727
--tserver_master_addrs=127.14.116.254:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:51:56.993112 31644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:51:56.994078 31644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:51:57.003364 31649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:57.003360 31652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:51:57.003378 31650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:51:57.003402 31644 server_base.cc:1034] running on GCE node
I20260619 06:51:57.004110 31644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:51:57.004773 31644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:51:57.005961 31644 hybrid_clock.cc:648] HybridClock initialized: now 1781851917005944 us; error 28 us; skew 500 ppm
Jun 19 06:51:57 dist-test-slave-9x3b krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851917, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:51:57.008853 31644 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:51:57.009671 31644 webserver.cc:460] Webserver started at http://127.14.116.195:33727/ using document root <none> and password file <none>
I20260619 06:51:57.010223 31644 fs_manager.cc:359] Metadata directory not provided
I20260619 06:51:57.010313 31644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:51:57.013908 31644 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:51:57.015941 31659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:57.017069 31644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:51:57.017195 31644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "1c7e8db0bfc2464d8694941a0b776da2"
format_stamp: "Formatted at 2026-06-19 06:51:15 on dist-test-slave-9x3b"
server_key: "3ecef243ed74ce300e236ad0a7372b7e"
server_key_iv: "a7493993a14a0ea48a21b96232557d1d"
server_key_version: "encryptionkey@0"
I20260619 06:51:57.017724 31644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:51:57.045053 31644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:51:57.047833 31644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:51:57.048023 31644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:51:57.048592 31644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:51:57.049364 31644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:51:57.049433 31644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:57.049510 31644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:51:57.049546 31644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:51:57.058033 31644 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:43867
I20260619 06:51:57.058076 31772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:43867 every 8 connection(s)
I20260619 06:51:57.058943 31644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:51:57.062549 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 31644
I20260619 06:51:57.063069 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:57.060839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:35235)
0619 06:51:57.061434 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:57.062016 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":145,"threads_started":1}
W20260619 06:51:57.064023 31773 heartbeater.cc:644] Failed to heartbeat to 127.14.116.254:38983 (0 consecutive failures): Network error: Failed to ping master at 127.14.116.254:38983: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
I20260619 06:51:57.064710 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:57.064414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:48267)
0619 06:51:57.064572 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:57.064632 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 06:51:57.065426 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:57.065187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:48565)
0619 06:51:57.065296 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:57.065363 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 06:51:57.065649 31773 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:51:57.694635 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:57.694222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:58475)
0619 06:51:57.694474 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:57.694527 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":67,"threads_started":1}
I20260619 06:51:57.874589 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:57.874114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:52257)
0619 06:51:57.874408 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:57.874476 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260619 06:51:58.066569 31779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:58.066191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:49283)
0619 06:51:58.066426 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:58.066476 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260619 06:51:58.695775 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:58.695340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:50537)
0619 06:51:58.695575 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:58.695678 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260619 06:51:58.875959 31781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:58.875443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:37875)
0619 06:51:58.875746 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:58.875817 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
I20260619 06:51:59.068486 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:59.067239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:33135)
0619 06:51:59.068322 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:59.068381 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1009,"thread_start_us":73,"threads_started":1}
I20260619 06:51:59.696779 31783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:59.696402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:58999)
0619 06:51:59.696619 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:59.696670 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260619 06:51:59.876994 31784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:51:59.876654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:53623)
0619 06:51:59.876860 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 06:51:59.876915 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260619 06:52:00.069542 31785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:00.069122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:40243)
0619 06:52:00.069388 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:00.069447 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260619 06:52:00.697960 31786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:00.697523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:58945)
0619 06:52:00.697797 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:00.697857 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260619 06:52:00.878283 31787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:00.877852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:40333)
0619 06:52:00.878122 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:00.878180 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260619 06:52:01.070788 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:01.070359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:52079)
0619 06:52:01.070610 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:01.070676 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260619 06:52:01.699395 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:01.698932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:38257)
0619 06:52:01.699230 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:01.699295 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260619 06:52:01.879487 31790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:01.879077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:36921)
0619 06:52:01.879329 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:01.879385 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260619 06:52:02.072094 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:02.071600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:43213)
0619 06:52:02.071940 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:02.072000 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":100,"threads_started":1}
I20260619 06:52:02.700583 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:02.700118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:41581)
0619 06:52:02.700427 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:02.700480 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":85,"threads_started":1}
I20260619 06:52:02.880635 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:02.880260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:40999)
0619 06:52:02.880505 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:02.880555 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":104,"threads_started":1}
I20260619 06:52:03.073174 31794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:03.072759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:51573)
0619 06:52:03.073031 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:03.073082 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260619 06:52:03.701741 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:03.701267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:57091)
0619 06:52:03.701591 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:03.701645 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":97,"threads_started":1}
I20260619 06:52:03.881767 31796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:03.881320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:36579)
0619 06:52:03.881606 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:03.881674 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":68,"threads_started":1}
I20260619 06:52:04.074321 31797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:04.073878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:47191)
0619 06:52:04.074163 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:04.074216 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
I20260619 06:52:04.702823 31798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:04.702440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:35039)
0619 06:52:04.702677 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:04.702729 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260619 06:52:04.882905 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:04.882465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:52525)
0619 06:52:04.882710 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:04.882763 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260619 06:52:05.075436 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:05.075020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:43681)
0619 06:52:05.075286 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:05.075347 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260619 06:52:05.704209 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:05.703813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:35165)
0619 06:52:05.704044 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:05.704118 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260619 06:52:05.884222 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:05.883812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:33881)
0619 06:52:05.884065 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:05.884124 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260619 06:52:06.076517 31803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:06.076107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:37435)
0619 06:52:06.076373 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:06.076428 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260619 06:52:06.705404 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:06.705009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:59163)
0619 06:52:06.705253 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:06.705308 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260619 06:52:06.885533 31805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:06.885114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:52923)
0619 06:52:06.885378 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:06.885437 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":70,"threads_started":1}
I20260619 06:52:07.077600 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:07.077228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:40105)
0619 06:52:07.077458 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:07.077510 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260619 06:52:07.706568 31807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:07.706149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:51857)
0619 06:52:07.706411 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:07.706476 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
I20260619 06:52:07.886647 31808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:07.886247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:44381)
0619 06:52:07.886490 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:07.886540 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
I20260619 06:52:08.078715 31809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:08.078339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:45185)
0619 06:52:08.078578 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:08.078628 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260619 06:52:08.707813 31810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:08.707295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:56845)
0619 06:52:08.707665 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:08.707718 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260619 06:52:08.887859 31811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:08.887371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:54509)
0619 06:52:08.887706 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:08.887763 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260619 06:52:09.079906 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:09.079508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:41841)
0619 06:52:09.079770 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:09.079821 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":66,"threads_started":1}
I20260619 06:52:09.709033 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:09.708544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:60581)
0619 06:52:09.708861 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:09.708940 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
I20260619 06:52:09.888976 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:09.888604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:52595)
0619 06:52:09.888832 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:09.888887 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260619 06:52:10.081236 31815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:10.080804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:47433)
0619 06:52:10.081078 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:10.081142 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260619 06:52:10.710265 31816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:10.709830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:52189)
0619 06:52:10.710104 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:10.710169 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260619 06:52:10.890475 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:10.889941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:41043)
0619 06:52:10.890296 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:10.890376 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":82,"threads_started":1}
I20260619 06:52:11.082500 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:11.082076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:39335)
0619 06:52:11.082332 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:11.082406 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260619 06:52:11.711669 31819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:11.711185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.193:39063)
0619 06:52:11.711456 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:11.711516 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":71,"threads_started":1}
I20260619 06:52:11.891930 31820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:11.891470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.194:56377)
0619 06:52:11.891778 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:11.891838 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":71,"threads_started":1}
I20260619 06:52:12.075280 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 31339
I20260619 06:52:12.080575 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 31512
I20260619 06:52:12.083777 31821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:12.083348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38983 (local address 127.14.116.195:53671)
0619 06:52:12.083590 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:12.083690 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.254:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260619 06:52:12.085950 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 31644
2026-06-19T06:52:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58984 ms)
[----------] 30 tests from SecurityITest (236671 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:12 dist-test-slave-9x3b krb5kdc[31824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:12 dist-test-slave-9x3b krb5kdc[31824](info): set up 2 sockets
Jun 19 06:52:12 dist-test-slave-9x3b krb5kdc[31824](info): commencing operation
krb5kdc: starting...
W20260619 06:52:13.123396 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:13 dist-test-slave-9x3b krb5kdc[31824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:13.235795 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:35461
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:35461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:13.340440 31837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:13.340719 31837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:13.340786 31837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:13.344694 31837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:13.344779 31837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:13.344818 31837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:13.344864 31837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:13.344900 31837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:13.349900 31837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:35461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:35461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:13.350834 31837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:13.351819 31837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:13.359889 31845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:13.360010 31843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:13.359923 31842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:13.359915 31837 server_base.cc:1034] running on GCE node
I20260619 06:52:13.360666 31837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:13.361605 31837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:13.362788 31837 hybrid_clock.cc:648] HybridClock initialized: now 1781851933362775 us; error 25 us; skew 500 ppm
Jun 19 06:52:13 dist-test-slave-9x3b krb5kdc[31824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:13.365507 31837 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:13.366333 31837 webserver.cc:460] Webserver started at http://127.14.116.254:39785/ using document root <none> and password file <none>
I20260619 06:52:13.366860 31837 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:13.366947 31837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:13.367163 31837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:13.369129 31837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "1112ff7665cd4a95aa7cf106307e0d84"
format_stamp: "Formatted at 2026-06-19 06:52:13 on dist-test-slave-9x3b"
server_key: "7b1f76d89ce7d44df3bb4dd39ab06e39"
server_key_iv: "017972324c0079126157c3b43ca4a3c7"
server_key_version: "encryptionkey@0"
I20260619 06:52:13.369671 31837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "1112ff7665cd4a95aa7cf106307e0d84"
format_stamp: "Formatted at 2026-06-19 06:52:13 on dist-test-slave-9x3b"
server_key: "7b1f76d89ce7d44df3bb4dd39ab06e39"
server_key_iv: "017972324c0079126157c3b43ca4a3c7"
server_key_version: "encryptionkey@0"
I20260619 06:52:13.373222 31837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 06:52:13.375461 31852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:13.376621 31837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:52:13.376756 31837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "1112ff7665cd4a95aa7cf106307e0d84"
format_stamp: "Formatted at 2026-06-19 06:52:13 on dist-test-slave-9x3b"
server_key: "7b1f76d89ce7d44df3bb4dd39ab06e39"
server_key_iv: "017972324c0079126157c3b43ca4a3c7"
server_key_version: "encryptionkey@0"
I20260619 06:52:13.376863 31837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:13.390664 31837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:13.393854 31837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:13.394048 31837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:13.400812 31837 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:35461
I20260619 06:52:13.400838 31904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:35461 every 8 connection(s)
I20260619 06:52:13.401696 31837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:13.404273 31905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:13.409457 31905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84: Bootstrap starting.
I20260619 06:52:13.410952 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 31837
I20260619 06:52:13.411110 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:13.411386 14803 external_mini_cluster.cc:1428] Setting key 51355cf2b6cdfe67d99167f9b09a4413
I20260619 06:52:13.411914 31905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:13.412714 31905 log.cc:824] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:13.414511 31905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84: No bootstrap required, opened a new log
I20260619 06:52:13.417701 31905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112ff7665cd4a95aa7cf106307e0d84" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35461 } }
I20260619 06:52:13.417922 31905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:13.417994 31905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1112ff7665cd4a95aa7cf106307e0d84, State: Initialized, Role: FOLLOWER
I20260619 06:52:13.418506 31905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112ff7665cd4a95aa7cf106307e0d84" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35461 } }
I20260619 06:52:13.418665 31905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:13.418736 31905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:13.418829 31905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:13.419770 31905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112ff7665cd4a95aa7cf106307e0d84" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35461 } }
Jun 19 06:52:13 dist-test-slave-9x3b krb5kdc[31824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:13.420113 31905 leader_election.cc:302] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1112ff7665cd4a95aa7cf106307e0d84; no voters:
I20260619 06:52:13.420472 31905 leader_election.cc:288] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:13.420560 31910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:13.420779 31910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [term 1 LEADER]: Becoming Leader. State: Replica: 1112ff7665cd4a95aa7cf106307e0d84, State: Running, Role: LEADER
I20260619 06:52:13.421105 31910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112ff7665cd4a95aa7cf106307e0d84" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35461 } }
I20260619 06:52:13.421710 31905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:13.422840 31912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1112ff7665cd4a95aa7cf106307e0d84. Latest consensus state: current_term: 1 leader_uuid: "1112ff7665cd4a95aa7cf106307e0d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112ff7665cd4a95aa7cf106307e0d84" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35461 } } }
I20260619 06:52:13.422995 31912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:13.422840 31911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1112ff7665cd4a95aa7cf106307e0d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112ff7665cd4a95aa7cf106307e0d84" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35461 } } }
I20260619 06:52:13.423327 31911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:13.423364 31919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:13.426159 31919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:13.427279 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:13.412536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.14.116.254:35461)
0619 06:52:13.412990 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:13.413005 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:13.413061 (+ 56us) server_negotiation.cc:407] Connection header received
0619 06:52:13.413856 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:13.413874 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:13.414184 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:13.414521 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:13.415331 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.416371 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.417229 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.417684 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.417859 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:13.420738 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:13.420744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:13.420760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:13.420789 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:13.424109 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:13.424655 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:13.424662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:13.424667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:13.424796 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:13.425122 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:13.425125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:13.425127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:13.425580 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:13.425775 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:13.426045 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:52:13.426338 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":126,"threads_started":1}
I20260619 06:52:13.431277 31919 catalog_manager.cc:1347] Generated new cluster ID: dd18e7bfff9b4f1688b6d8510e1d9e64
I20260619 06:52:13.431339 31919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:13.436574 31919 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:13.437355 31919 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:13.447125 31919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84: Generated new TSK 0
I20260619 06:52:13.447801 31919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:52:13.478484 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35461
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:13.606114 31931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:13.606388 31931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:13.606467 31931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:13.610507 31931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:13.610591 31931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:13.610684 31931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:52:13.615891 31931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:13.616855 31931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:13.617815 31931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:13.627002 31936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:13.627008 31939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:13.627183 31931 server_base.cc:1034] running on GCE node
W20260619 06:52:13.627010 31937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:13.627719 31931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:13.628350 31931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:13.629508 31931 hybrid_clock.cc:648] HybridClock initialized: now 1781851933629494 us; error 25 us; skew 500 ppm
Jun 19 06:52:13 dist-test-slave-9x3b krb5kdc[31824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:13.632187 31931 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:52:13.632978 31931 webserver.cc:460] Webserver started at http://127.14.116.193:33319/ using document root <none> and password file <none>
I20260619 06:52:13.633502 31931 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:13.633558 31931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:13.633822 31931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:13.635855 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "5bd349b0d8fc4f9fae7056ebf5002e62"
format_stamp: "Formatted at 2026-06-19 06:52:13 on dist-test-slave-9x3b"
server_key: "ee1f17c5f2616a1a8b912e542752c372"
server_key_iv: "f45d0a363b4fd50ddaaff47e7a3b647d"
server_key_version: "encryptionkey@0"
I20260619 06:52:13.636381 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "5bd349b0d8fc4f9fae7056ebf5002e62"
format_stamp: "Formatted at 2026-06-19 06:52:13 on dist-test-slave-9x3b"
server_key: "ee1f17c5f2616a1a8b912e542752c372"
server_key_iv: "f45d0a363b4fd50ddaaff47e7a3b647d"
server_key_version: "encryptionkey@0"
I20260619 06:52:13.639827 31931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:52:13.642037 31946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:13.643172 31931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:13.643296 31931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "5bd349b0d8fc4f9fae7056ebf5002e62"
format_stamp: "Formatted at 2026-06-19 06:52:13 on dist-test-slave-9x3b"
server_key: "ee1f17c5f2616a1a8b912e542752c372"
server_key_iv: "f45d0a363b4fd50ddaaff47e7a3b647d"
server_key_version: "encryptionkey@0"
I20260619 06:52:13.643393 31931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:13.654546 31931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:13.657595 31931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:13.657788 31931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:13.658308 31931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:13.659063 31931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:13.659107 31931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:13.659170 31931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:13.659202 31931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:13.667488 31931 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:34589
I20260619 06:52:13.667514 32059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:34589 every 8 connection(s)
I20260619 06:52:13.668330 31931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:52:13.674643 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 31931
I20260619 06:52:13.674819 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:52:13.675104 14803 external_mini_cluster.cc:1428] Setting key c4353defd84b4030a1bb047e0d78e958
Jun 19 06:52:13 dist-test-slave-9x3b krb5kdc[31824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:13.682132 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:13.669941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:48529 (local address 127.14.116.254:35461)
0619 06:52:13.670076 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:13.670079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:13.670856 (+ 777us) server_negotiation.cc:407] Connection header received
0619 06:52:13.672211 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:13.672214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:13.672271 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:13.672376 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:13.673455 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.674019 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.675321 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.675687 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.675795 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:13.678337 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:13.678341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:13.678342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:13.678374 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:13.679924 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:13.680552 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:13.680555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:13.680557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:13.680629 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:13.681048 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:13.681050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:13.681051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:13.681213 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:13.681286 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:13.681918 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 06:52:13.682018 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:52:13.683342 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:13.670289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35461 (local address 127.14.116.193:48529)
0619 06:52:13.670706 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:13.670745 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:13.672017 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:13.672539 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:13.672545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:13.672861 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:13.673296 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:13.673307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.674145 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.674148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:13.675151 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:13.675160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.675852 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.675856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:13.676021 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:13.676649 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:13.676664 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:13.678158 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:13.680040 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:13.680046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:13.680062 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:13.680443 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:13.680750 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:13.680756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:13.680761 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:13.680943 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:13.681317 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:13.681333 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:13.681661 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 06:52:13.681890 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 06:52:13.682304 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
I20260619 06:52:13.684633 32060 heartbeater.cc:347] Connected to a master server at 127.14.116.254:35461
I20260619 06:52:13.684885 32060 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:13.685351 32060 heartbeater.cc:511] Master 127.14.116.254:35461 requested a full tablet report, sending...
I20260619 06:52:13.686810 31869 ts_manager.cc:194] Registered new tserver with Master: 5bd349b0d8fc4f9fae7056ebf5002e62 (127.14.116.193:34589)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
I20260619 06:52:13.688001 31869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:48529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:52:13.700575 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35461
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:13.805217 32065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:13.805456 32065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:13.805498 32065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:13.809505 32065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:13.809556 32065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:13.809612 32065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:52:13.814538 32065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:13.815490 32065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:13.816418 32065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:13.825603 32070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:13.825603 32073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:13.825677 32071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:13.825874 32065 server_base.cc:1034] running on GCE node
I20260619 06:52:13.826315 32065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:13.826936 32065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:13.828091 32065 hybrid_clock.cc:648] HybridClock initialized: now 1781851933828073 us; error 29 us; skew 500 ppm
Jun 19 06:52:13 dist-test-slave-9x3b krb5kdc[31824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:13.830667 32065 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:52:13.831476 32065 webserver.cc:460] Webserver started at http://127.14.116.194:43777/ using document root <none> and password file <none>
I20260619 06:52:13.832059 32065 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:13.832118 32065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:13.832340 32065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:13.834220 32065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "1da5233021094d45bc6593559b1a45b0"
format_stamp: "Formatted at 2026-06-19 06:52:13 on dist-test-slave-9x3b"
server_key: "3ea4b1c2b8da9c1cb38cc7bf7ad9f0df"
server_key_iv: "d3f0a58b98da5b483f78869c58abbc80"
server_key_version: "encryptionkey@0"
I20260619 06:52:13.834762 32065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "1da5233021094d45bc6593559b1a45b0"
format_stamp: "Formatted at 2026-06-19 06:52:13 on dist-test-slave-9x3b"
server_key: "3ea4b1c2b8da9c1cb38cc7bf7ad9f0df"
server_key_iv: "d3f0a58b98da5b483f78869c58abbc80"
server_key_version: "encryptionkey@0"
I20260619 06:52:13.838160 32065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 06:52:13.840377 32080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:13.841498 32065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:52:13.841624 32065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "1da5233021094d45bc6593559b1a45b0"
format_stamp: "Formatted at 2026-06-19 06:52:13 on dist-test-slave-9x3b"
server_key: "3ea4b1c2b8da9c1cb38cc7bf7ad9f0df"
server_key_iv: "d3f0a58b98da5b483f78869c58abbc80"
server_key_version: "encryptionkey@0"
I20260619 06:52:13.841727 32065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:13.859329 32065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:13.861991 32065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:13.862178 32065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:13.862709 32065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:13.863463 32065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:13.863531 32065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:13.863600 32065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:13.863672 32065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:13.872035 32065 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:36253
I20260619 06:52:13.872133 32193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:36253 every 8 connection(s)
I20260619 06:52:13.872951 32065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:52:13.875840 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 32065
I20260619 06:52:13.875993 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:52:13.876407 14803 external_mini_cluster.cc:1428] Setting key 148e9be892f0b63699a6ed9550f3daf5
Jun 19 06:52:13 dist-test-slave-9x3b krb5kdc[31824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:52:13.888000 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:13.874632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:45915 (local address 127.14.116.254:35461)
0619 06:52:13.874778 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:13.874783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:13.875583 (+ 800us) server_negotiation.cc:407] Connection header received
0619 06:52:13.877573 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:13.877579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:13.877623 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:13.877698 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:13.878821 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.879589 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.881405 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.881619 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.881695 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:13.884523 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:13.884526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:13.884529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:13.884551 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:13.885966 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:13.886498 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:13.886501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:13.886503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:13.886574 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:13.886906 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:13.886910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:13.886912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:13.887089 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:13.887196 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:13.887699 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 06:52:13.887819 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:52:13.888794 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:13.875018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35461 (local address 127.14.116.194:45915)
0619 06:52:13.875433 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:13.875472 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:13.876969 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:13.877894 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:13.877900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:13.878221 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:13.878673 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:13.878686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.879769 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.879775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:13.881263 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:13.881277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:13.881771 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:13.881778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:13.882007 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:13.882575 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:13.882600 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:13.884334 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:13.886082 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:13.886088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:13.886101 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:13.886382 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:13.886664 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:13.886669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:13.886671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:13.886807 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:13.887193 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:13.887199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:13.887443 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 06:52:13.887646 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 06:52:13.887955 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260619 06:52:13.890023 32194 heartbeater.cc:347] Connected to a master server at 127.14.116.254:35461
I20260619 06:52:13.890260 32194 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:13.890714 32194 heartbeater.cc:511] Master 127.14.116.254:35461 requested a full tablet report, sending...
I20260619 06:52:13.891793 31869 ts_manager.cc:194] Registered new tserver with Master: 1da5233021094d45bc6593559b1a45b0 (127.14.116.194:36253)
I20260619 06:52:13.892344 31869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:45915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:52:13.903805 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35461
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:38733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:14.012023 32199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:14.012265 32199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:14.012306 32199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:14.016309 32199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:14.016366 32199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:14.016479 32199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:52:14.021723 32199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:38733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:14.022783 32199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:14.023813 32199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:14.032985 32204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:14.032992 32207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:14.033128 32205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:14.033389 32199 server_base.cc:1034] running on GCE node
I20260619 06:52:14.033775 32199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:14.034368 32199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:14.035521 32199 hybrid_clock.cc:648] HybridClock initialized: now 1781851934035509 us; error 24 us; skew 500 ppm
Jun 19 06:52:14 dist-test-slave-9x3b krb5kdc[31824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:14.038148 32199 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:52:14.039016 32199 webserver.cc:460] Webserver started at http://127.14.116.195:36579/ using document root <none> and password file <none>
I20260619 06:52:14.039541 32199 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:14.039598 32199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:14.039863 32199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:14.041781 32199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "d3b78ed8780e2c0308bda8e1342f0e7c"
server_key_iv: "df881660c8cbfd72c7cd2202bbf7781e"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.042346 32199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "d3b78ed8780e2c0308bda8e1342f0e7c"
server_key_iv: "df881660c8cbfd72c7cd2202bbf7781e"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.046185 32199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 06:52:14.048664 32214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.049834 32199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:52:14.049952 32199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "d3b78ed8780e2c0308bda8e1342f0e7c"
server_key_iv: "df881660c8cbfd72c7cd2202bbf7781e"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.050058 32199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:14.063099 32199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:14.065713 32199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:14.065896 32199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:14.066428 32199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:14.067168 32199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:14.067219 32199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.067286 32199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:14.067325 32199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.076013 32199 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:34249
I20260619 06:52:14.076037 32327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:34249 every 8 connection(s)
I20260619 06:52:14.076833 32199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:52:14.078894 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 32199
I20260619 06:52:14.079033 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:52:14.079332 14803 external_mini_cluster.cc:1428] Setting key f99da4f252240629229782cb1e052456
Jun 19 06:52:14 dist-test-slave-9x3b krb5kdc[31824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:14.090020 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.078501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:50151 (local address 127.14.116.254:35461)
0619 06:52:14.078640 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.078644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.079460 (+ 816us) server_negotiation.cc:407] Connection header received
0619 06:52:14.081090 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.081094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.081149 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.081253 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:14.082470 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.082992 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.084199 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.084429 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.084539 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.086772 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:14.086782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:14.086785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:14.086807 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:14.088098 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:14.088586 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:14.088591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:14.088593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:14.088671 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:14.089004 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:14.089009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:14.089011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:14.089180 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:14.089288 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.089739 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.089855 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:52:14.090736 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.078840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:35461 (local address 127.14.116.195:50151)
0619 06:52:14.079315 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:14.079366 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:14.080892 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:14.081546 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:14.081552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:14.081889 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:14.082329 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.082342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.083112 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.083116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.084075 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.084085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.084571 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.084575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.084739 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.085230 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:14.085245 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:14.086615 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:14.088212 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:14.088217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:14.088230 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:14.088483 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:14.088762 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:14.088767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:14.088769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:14.088908 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:14.089284 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:14.089289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:14.089519 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 06:52:14.089700 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 06:52:14.090003 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":170,"threads_started":1}
I20260619 06:52:14.091902 32328 heartbeater.cc:347] Connected to a master server at 127.14.116.254:35461
I20260619 06:52:14.092146 32328 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:14.092725 32328 heartbeater.cc:511] Master 127.14.116.254:35461 requested a full tablet report, sending...
I20260619 06:52:14.093688 31869 ts_manager.cc:194] Registered new tserver with Master: 8585e0f4ab0843a9abd65aee9a6dfe7c (127.14.116.195:34249)
I20260619 06:52:14.094201 31869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:50151
I20260619 06:52:14.104734 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:52:14.113047 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.106560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.14.116.254:35461)
0619 06:52:14.106673 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.106677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.106842 (+ 165us) server_negotiation.cc:407] Connection header received
0619 06:52:14.106950 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.106952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.106986 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.107043 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:14.107699 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.108195 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.108959 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.109154 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.109249 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.110514 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:14.110517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:14.110519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:14.110534 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:14.111752 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:14.112116 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:14.112118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:14.112120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:14.112186 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:14.112399 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:14.112401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:14.112402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:14.112591 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:14.112690 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.112857 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.112939 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:52:14.115499 31869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:52:14.118462 31869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:52:14.132241 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.127019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.14.116.194:36253)
0619 06:52:14.127531 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.127546 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.127789 (+ 243us) server_negotiation.cc:407] Connection header received
0619 06:52:14.127917 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.127922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.128143 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.128297 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:14.128594 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.129511 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.130731 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.131567 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.131734 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.131886 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.132012 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.132103 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":90,"threads_started":1}
I20260619 06:52:14.132246 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.126869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:36253 (local address 127.0.0.1:51248)
0619 06:52:14.127688 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:14.127698 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:14.127804 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:14.128298 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:14.128303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:14.128325 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:14.128464 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.128470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.129641 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.129644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.130598 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.130606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.131709 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.131713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.131821 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.131862 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:52:14.131978 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 06:52:14.132098 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260619 06:52:14.132799 32340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.126788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.14.116.193:34589)
0619 06:52:14.127669 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.127681 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.127704 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:52:14.127779 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.127784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.128003 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.128162 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:14.128674 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.129834 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.131242 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.131988 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.132090 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.132200 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.132538 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.132655 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":138,"threads_started":1}
I20260619 06:52:14.133158 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.126572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:34589 (local address 127.0.0.1:56912)
0619 06:52:14.127299 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:14.127348 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:14.127474 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:14.128448 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:14.128451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:14.128465 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:14.128564 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.128570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.129981 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.129984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.131141 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.131151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.132259 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.132266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.132378 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.132400 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:52:14.132942 (+ 542us) client_negotiation.cc:240] Negotiation successful
0619 06:52:14.133039 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":23,"thread_start_us":63,"threads_started":1}
I20260619 06:52:14.133371 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.127413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34249 (local address 127.0.0.1:41548)
0619 06:52:14.128405 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:14.128419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:14.128505 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:14.129027 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:14.129030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:14.129048 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:14.129127 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.129132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.130454 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.130457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.131596 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.131647 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.132895 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.132900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.133040 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.133064 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:52:14.133158 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:52:14.133231 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260619 06:52:14.133522 32344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.127847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.14.116.195:34249)
0619 06:52:14.128180 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.128190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.128601 (+ 411us) server_negotiation.cc:407] Connection header received
0619 06:52:14.128695 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.128699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.128878 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.129024 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:14.129243 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.130315 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.131760 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.132601 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.132721 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.132817 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.133203 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.133369 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260619 06:52:14.136176 32262 tablet_service.cc:1467] Processing CreateTablet for tablet 4b711220ef464f4791729773dc1edc63 (DEFAULT_TABLE table=test-table [id=ad895484b3994bd7971164a7a9fd30b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:14.136173 32128 tablet_service.cc:1467] Processing CreateTablet for tablet 4b711220ef464f4791729773dc1edc63 (DEFAULT_TABLE table=test-table [id=ad895484b3994bd7971164a7a9fd30b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:14.137255 32128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b711220ef464f4791729773dc1edc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:14.137255 32262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b711220ef464f4791729773dc1edc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:14.137089 31994 tablet_service.cc:1467] Processing CreateTablet for tablet 4b711220ef464f4791729773dc1edc63 (DEFAULT_TABLE table=test-table [id=ad895484b3994bd7971164a7a9fd30b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:14.137970 31994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b711220ef464f4791729773dc1edc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:14.143728 32345 tablet_bootstrap.cc:492] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62: Bootstrap starting.
I20260619 06:52:14.144932 32347 tablet_bootstrap.cc:492] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: Bootstrap starting.
I20260619 06:52:14.145952 32345 tablet_bootstrap.cc:654] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:14.146852 32347 tablet_bootstrap.cc:654] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:14.146972 32346 tablet_bootstrap.cc:492] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: Bootstrap starting.
I20260619 06:52:14.147212 32345 log.cc:824] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:14.148098 32347 log.cc:824] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:14.149071 32346 tablet_bootstrap.cc:654] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:14.149765 32345 tablet_bootstrap.cc:492] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62: No bootstrap required, opened a new log
I20260619 06:52:14.149955 32345 ts_tablet_manager.cc:1397] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:52:14.150199 32346 log.cc:824] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:14.151981 32347 tablet_bootstrap.cc:492] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: No bootstrap required, opened a new log
I20260619 06:52:14.151980 32346 tablet_bootstrap.cc:492] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: No bootstrap required, opened a new log
I20260619 06:52:14.152146 32346 ts_tablet_manager.cc:1397] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:52:14.152146 32347 ts_tablet_manager.cc:1397] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 06:52:14.153164 32345 raft_consensus.cc:348] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } }
I20260619 06:52:14.153391 32345 raft_consensus.cc:374] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:14.153461 32345 raft_consensus.cc:729] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd349b0d8fc4f9fae7056ebf5002e62, State: Initialized, Role: FOLLOWER
I20260619 06:52:14.153939 32345 consensus_queue.cc:260] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } }
I20260619 06:52:14.154556 32060 heartbeater.cc:503] Master 127.14.116.254:35461 was elected leader, sending a full tablet report...
I20260619 06:52:14.155670 32345 ts_tablet_manager.cc:1428] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:52:14.155905 32347 raft_consensus.cc:348] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } }
I20260619 06:52:14.155905 32346 raft_consensus.cc:348] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } }
I20260619 06:52:14.156138 32347 raft_consensus.cc:374] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:14.156138 32346 raft_consensus.cc:374] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:14.156193 32346 raft_consensus.cc:729] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8585e0f4ab0843a9abd65aee9a6dfe7c, State: Initialized, Role: FOLLOWER
I20260619 06:52:14.156219 32347 raft_consensus.cc:729] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da5233021094d45bc6593559b1a45b0, State: Initialized, Role: FOLLOWER
I20260619 06:52:14.156628 32346 consensus_queue.cc:260] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } }
I20260619 06:52:14.156628 32347 consensus_queue.cc:260] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } }
I20260619 06:52:14.157473 32328 heartbeater.cc:503] Master 127.14.116.254:35461 was elected leader, sending a full tablet report...
I20260619 06:52:14.157526 32346 ts_tablet_manager.cc:1428] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:52:14.157749 32347 ts_tablet_manager.cc:1428] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:52:14.158485 32194 heartbeater.cc:503] Master 127.14.116.254:35461 was elected leader, sending a full tablet report...
W20260619 06:52:14.169365 32061 tablet.cc:2369] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:52:14.173583 32352 raft_consensus.cc:482] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:52:14.173811 32352 raft_consensus.cc:504] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } }
I20260619 06:52:14.174888 32352 leader_election.cc:288] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bd349b0d8fc4f9fae7056ebf5002e62 (127.14.116.193:34589), 8585e0f4ab0843a9abd65aee9a6dfe7c (127.14.116.195:34249)
I20260619 06:52:14.179299 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.175094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:34589 (local address 127.14.116.194:50967)
0619 06:52:14.175242 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:14.175258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:14.175344 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:14.175659 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:14.175661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:14.175680 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:14.175810 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.175817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.176820 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.176823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.178039 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.178048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.178859 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.178861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.178975 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.178999 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:52:14.179077 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 06:52:14.179160 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:52:14.179338 32340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.175178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:50967 (local address 127.14.116.193:34589)
0619 06:52:14.175332 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.175337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.175353 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:52:14.175468 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.175474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.175530 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.175638 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:14.175924 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.176717 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.178158 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.178746 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.178862 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.178932 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.179106 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.179213 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:52:14.179332 32344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.175453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:33281 (local address 127.14.116.195:34249)
0619 06:52:14.175555 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.175560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.175829 (+ 269us) server_negotiation.cc:407] Connection header received
0619 06:52:14.175937 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.175940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.175988 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.176062 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:14.176286 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.176952 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.178268 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.178856 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.178959 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.179003 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.179163 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.179225 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 06:52:14.179307 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.175350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34249 (local address 127.14.116.194:33281)
0619 06:52:14.175731 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:14.175744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:14.175836 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:14.176083 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:14.176085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:14.176098 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:14.176188 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.176192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.177047 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.177050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.178170 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.178177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.178956 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.178958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.179049 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.179078 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:52:14.179141 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 06:52:14.179186 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":130,"threads_started":1}
I20260619 06:52:14.180019 32014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b711220ef464f4791729773dc1edc63" candidate_uuid: "1da5233021094d45bc6593559b1a45b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" is_pre_election: true
I20260619 06:52:14.180063 32282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b711220ef464f4791729773dc1edc63" candidate_uuid: "1da5233021094d45bc6593559b1a45b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" is_pre_election: true
I20260619 06:52:14.180330 32014 raft_consensus.cc:2454] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da5233021094d45bc6593559b1a45b0 in term 0.
I20260619 06:52:14.180320 32282 raft_consensus.cc:2454] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da5233021094d45bc6593559b1a45b0 in term 0.
I20260619 06:52:14.180740 32081 leader_election.cc:302] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da5233021094d45bc6593559b1a45b0, 8585e0f4ab0843a9abd65aee9a6dfe7c; no voters:
I20260619 06:52:14.181035 32352 raft_consensus.cc:2781] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:52:14.181118 32352 raft_consensus.cc:482] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:52:14.181149 32352 raft_consensus.cc:3037] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:14.182159 32352 raft_consensus.cc:504] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } }
I20260619 06:52:14.182497 32352 leader_election.cc:288] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [CANDIDATE]: Term 1 election: Requested vote from peers 5bd349b0d8fc4f9fae7056ebf5002e62 (127.14.116.193:34589), 8585e0f4ab0843a9abd65aee9a6dfe7c (127.14.116.195:34249)
I20260619 06:52:14.182868 32014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b711220ef464f4791729773dc1edc63" candidate_uuid: "1da5233021094d45bc6593559b1a45b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62"
I20260619 06:52:14.182992 32014 raft_consensus.cc:3037] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:14.183039 32282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b711220ef464f4791729773dc1edc63" candidate_uuid: "1da5233021094d45bc6593559b1a45b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c"
I20260619 06:52:14.183190 32282 raft_consensus.cc:3037] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:14.184082 32014 raft_consensus.cc:2454] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da5233021094d45bc6593559b1a45b0 in term 1.
I20260619 06:52:14.184413 32081 leader_election.cc:302] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da5233021094d45bc6593559b1a45b0, 5bd349b0d8fc4f9fae7056ebf5002e62; no voters:
I20260619 06:52:14.184556 32282 raft_consensus.cc:2454] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da5233021094d45bc6593559b1a45b0 in term 1.
I20260619 06:52:14.184574 32352 raft_consensus.cc:2781] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:14.184773 32352 raft_consensus.cc:686] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 1 LEADER]: Becoming Leader. State: Replica: 1da5233021094d45bc6593559b1a45b0, State: Running, Role: LEADER
I20260619 06:52:14.185094 32352 consensus_queue.cc:237] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } }
I20260619 06:52:14.188162 31868 catalog_manager.cc:5495] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1da5233021094d45bc6593559b1a45b0 (127.14.116.194). New cstate: current_term: 1 leader_uuid: "1da5233021094d45bc6593559b1a45b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1da5233021094d45bc6593559b1a45b0" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 } health_report { overall_health: UNKNOWN } } }
I20260619 06:52:14.209667 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.206070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.14.116.194:36253)
0619 06:52:14.206244 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.206249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.206262 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:52:14.206393 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.206395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.206444 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.206553 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:14.206789 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.207339 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.208256 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.208481 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.208591 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.208890 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:14.209279 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:14.209421 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.209460 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.209511 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:52:14.214915 32282 raft_consensus.cc:1261] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [term 1 FOLLOWER]: Refusing update from remote peer 1da5233021094d45bc6593559b1a45b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:14.214941 32014 raft_consensus.cc:1261] T 4b711220ef464f4791729773dc1edc63 P 5bd349b0d8fc4f9fae7056ebf5002e62 [term 1 FOLLOWER]: Refusing update from remote peer 1da5233021094d45bc6593559b1a45b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:14.215567 32352 consensus_queue.cc:1035] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd349b0d8fc4f9fae7056ebf5002e62" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:14.215749 32355 consensus_queue.cc:1035] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8585e0f4ab0843a9abd65aee9a6dfe7c" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:14.223875 32359 mvcc.cc:205] Tried to move back new op lower bound from 7298465522537889792 to 7298465522424786944. Current Snapshot: MvccSnapshot[applied={T|T < 7298465522537889792}]
I20260619 06:52:14.225203 32360 mvcc.cc:205] Tried to move back new op lower bound from 7298465522537889792 to 7298465522424786944. Current Snapshot: MvccSnapshot[applied={T|T < 7298465522537889792 or (T in {7298465522537889792})}]
I20260619 06:52:14.230371 31868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:14.230505 31868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:14.232774 31868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1112ff7665cd4a95aa7cf106307e0d84: Sending DeleteTablet for 3 replicas of tablet 4b711220ef464f4791729773dc1edc63
I20260619 06:52:14.233494 32128 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b711220ef464f4791729773dc1edc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:14 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:51248
I20260619 06:52:14.233668 31994 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b711220ef464f4791729773dc1edc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:14 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:56912
I20260619 06:52:14.234068 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 31931
I20260619 06:52:14.234098 32262 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b711220ef464f4791729773dc1edc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:14 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:41548
I20260619 06:52:14.234711 32370 tablet_replica.cc:330] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: stopping tablet replica
I20260619 06:52:14.235005 32370 raft_consensus.cc:2229] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:14.235330 32370 raft_consensus.cc:2258] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:14.235709 32368 tablet_replica.cc:330] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: stopping tablet replica
I20260619 06:52:14.236003 32368 raft_consensus.cc:2229] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:52:14.236399 32368 raft_consensus.cc:2258] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:14.237735 32368 ts_tablet_manager.cc:1905] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:14.237752 32370 ts_tablet_manager.cc:1905] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:14.240388 32370 ts_tablet_manager.cc:1918] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:14.240388 32368 ts_tablet_manager.cc:1918] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:14.240515 32370 log.cc:1196] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/4b711220ef464f4791729773dc1edc63
I20260619 06:52:14.240528 32368 log.cc:1196] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/4b711220ef464f4791729773dc1edc63
I20260619 06:52:14.240886 32370 ts_tablet_manager.cc:1939] T 4b711220ef464f4791729773dc1edc63 P 8585e0f4ab0843a9abd65aee9a6dfe7c: Deleting consensus metadata
I20260619 06:52:14.240963 32368 ts_tablet_manager.cc:1939] T 4b711220ef464f4791729773dc1edc63 P 1da5233021094d45bc6593559b1a45b0: Deleting consensus metadata
I20260619 06:52:14.242015 31853 catalog_manager.cc:4841] TS 8585e0f4ab0843a9abd65aee9a6dfe7c (127.14.116.195:34249): tablet 4b711220ef464f4791729773dc1edc63 (table test-table [id=ad895484b3994bd7971164a7a9fd30b7]) successfully deleted
I20260619 06:52:14.242105 31856 catalog_manager.cc:4841] TS 1da5233021094d45bc6593559b1a45b0 (127.14.116.194:36253): tablet 4b711220ef464f4791729773dc1edc63 (table test-table [id=ad895484b3994bd7971164a7a9fd30b7]) successfully deleted
W20260619 06:52:14.243870 31853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:34589 (error 108)
I20260619 06:52:14.243913 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 32065
I20260619 06:52:14.245831 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.245503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:34589 (local address 127.0.0.1:56914)
0619 06:52:14.245658 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:14.245736 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:34589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 06:52:14.245973 31853 catalog_manager.cc:4584] TS 5bd349b0d8fc4f9fae7056ebf5002e62 (127.14.116.193:34589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b711220ef464f4791729773dc1edc63: Network error: Client connection negotiation failed: client connection to 127.14.116.193:34589: connect: Connection refused (error 111)
I20260619 06:52:14.249725 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 32199
I20260619 06:52:14.254918 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 31837
2026-06-19T06:52:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2169 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T06:52:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:14Z Disabled control of system clock
I20260619 06:52:14.283907 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:34085
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:52:14.390406 32375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:14.390669 32375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:14.390703 32375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:14.394690 32375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:14.394747 32375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:14.394764 32375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:14.394781 32375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:14.394796 32375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:14.399573 32375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:34085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:14.400631 32375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:14.401965 32375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:14.410072 32380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:14.410072 32381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:14.410096 32383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:14.410306 32375 server_base.cc:1034] running on GCE node
I20260619 06:52:14.410928 32375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:14.412011 32375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:14.413198 32375 hybrid_clock.cc:648] HybridClock initialized: now 1781851934413184 us; error 23 us; skew 500 ppm
I20260619 06:52:14.414963 32375 webserver.cc:460] Webserver started at http://127.14.116.254:46643/ using document root <none> and password file <none>
I20260619 06:52:14.415504 32375 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:14.415592 32375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:14.415813 32375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:14.417728 32375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "d0de89668941420a91a2e185f974dee3"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "153a33c8f1117b24ef8a79cf436f8a60"
server_key_iv: "ddae474e990f90abd7f58d062ec6222f"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.418256 32375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "d0de89668941420a91a2e185f974dee3"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "153a33c8f1117b24ef8a79cf436f8a60"
server_key_iv: "ddae474e990f90abd7f58d062ec6222f"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.421633 32375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 06:52:14.423928 32389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.425043 32375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 06:52:14.425173 32375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "d0de89668941420a91a2e185f974dee3"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "153a33c8f1117b24ef8a79cf436f8a60"
server_key_iv: "ddae474e990f90abd7f58d062ec6222f"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.425275 32375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:14.449139 32375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:14.449826 32375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:14.450013 32375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:14.456537 32375 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:34085
I20260619 06:52:14.456550 32441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:34085 every 8 connection(s)
I20260619 06:52:14.457458 32375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:14.459188 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 32375
I20260619 06:52:14.459326 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:14.459594 14803 external_mini_cluster.cc:1428] Setting key 3f1019e2db3b510ec5a053e56945a04a
I20260619 06:52:14.460685 32442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:14.467114 32442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3: Bootstrap starting.
I20260619 06:52:14.469669 32442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:14.469821 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.460992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.14.116.254:34085)
0619 06:52:14.461628 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.461648 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.461688 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:52:14.462582 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.462604 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.463085 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.463452 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:14.463893 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.465298 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.466218 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.466870 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.467034 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.467861 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:14.467867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:14.467882 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:14.468390 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:14.468456 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:14.468482 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:14.468577 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.468698 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.468941 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260619 06:52:14.470444 32442 log.cc:824] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:14.472245 32442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3: No bootstrap required, opened a new log
I20260619 06:52:14.475531 32442 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0de89668941420a91a2e185f974dee3" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34085 } }
I20260619 06:52:14.475802 32442 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:14.475898 32442 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0de89668941420a91a2e185f974dee3, State: Initialized, Role: FOLLOWER
I20260619 06:52:14.476295 32442 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0de89668941420a91a2e185f974dee3" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34085 } }
I20260619 06:52:14.476444 32442 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:14.476526 32442 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:14.476639 32442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:14.477433 32442 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0de89668941420a91a2e185f974dee3" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34085 } }
I20260619 06:52:14.477756 32442 leader_election.cc:302] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0de89668941420a91a2e185f974dee3; no voters:
I20260619 06:52:14.478044 32442 leader_election.cc:288] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:14.478170 32447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:14.478391 32447 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [term 1 LEADER]: Becoming Leader. State: Replica: d0de89668941420a91a2e185f974dee3, State: Running, Role: LEADER
I20260619 06:52:14.478657 32447 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0de89668941420a91a2e185f974dee3" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34085 } }
I20260619 06:52:14.479178 32442 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:14.480340 32448 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0de89668941420a91a2e185f974dee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0de89668941420a91a2e185f974dee3" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34085 } } }
I20260619 06:52:14.480363 32449 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0de89668941420a91a2e185f974dee3. Latest consensus state: current_term: 1 leader_uuid: "d0de89668941420a91a2e185f974dee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0de89668941420a91a2e185f974dee3" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 34085 } } }
I20260619 06:52:14.480496 32448 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:14.480496 32449 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:14.480913 32456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:14.483420 32456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:14.488564 32456 catalog_manager.cc:1347] Generated new cluster ID: ffc0713654c048649db6c287614a8004
I20260619 06:52:14.488626 32456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:14.504312 32456 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:14.505553 32456 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:14.514444 32456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3: Generated new TSK 0
I20260619 06:52:14.515100 32456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:14.518863 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34085
--builtin_ntp_servers=127.14.116.212:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:52:14.628153 32466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:14.628419 32466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:14.628491 32466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:14.632588 32466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:14.632658 32466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:14.632740 32466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:52:14.637463 32466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:14.638432 32466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:14.639727 32466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:14.648612 32471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:14.648617 32472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:14.648723 32474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:14.648993 32466 server_base.cc:1034] running on GCE node
I20260619 06:52:14.649391 32466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:14.649971 32466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:14.651173 32466 hybrid_clock.cc:648] HybridClock initialized: now 1781851934651150 us; error 57 us; skew 500 ppm
I20260619 06:52:14.652961 32466 webserver.cc:460] Webserver started at http://127.14.116.193:42275/ using document root <none> and password file <none>
I20260619 06:52:14.653513 32466 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:14.653604 32466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:14.653826 32466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:14.655828 32466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "e9f51565fa0e4776a923b7e95d70829f"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "04a01c45a0faea76bc15102c6a0d0e04"
server_key_iv: "b2492d3ca3f5b28133060486e38ce9d9"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.656378 32466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "e9f51565fa0e4776a923b7e95d70829f"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "04a01c45a0faea76bc15102c6a0d0e04"
server_key_iv: "b2492d3ca3f5b28133060486e38ce9d9"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.659799 32466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:52:14.662003 32480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.663132 32466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:14.663291 32466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "e9f51565fa0e4776a923b7e95d70829f"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "04a01c45a0faea76bc15102c6a0d0e04"
server_key_iv: "b2492d3ca3f5b28133060486e38ce9d9"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.663399 32466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:14.680656 32466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:14.681327 32466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:14.681565 32466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:14.682150 32466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:14.683040 32466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:14.683109 32466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.683172 32466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:14.683221 32466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.691891 32466 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:44123
I20260619 06:52:14.691913 32593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:44123 every 8 connection(s)
I20260619 06:52:14.692806 32466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:52:14.694886 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 32466
I20260619 06:52:14.695019 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:52:14.695363 14803 external_mini_cluster.cc:1428] Setting key 2e8a366f8ad0c05c963f3a064027242e
I20260619 06:52:14.697379 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34085
--builtin_ntp_servers=127.14.116.212:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:52:14.702862 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.694574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:34197 (local address 127.14.116.254:34085)
0619 06:52:14.694810 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.694814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.696061 (+ 1247us) server_negotiation.cc:407] Connection header received
0619 06:52:14.697620 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.697626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.697687 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.697786 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:14.698763 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.699276 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.700531 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.700944 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.701078 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.702265 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:14.702271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:14.702276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:14.702300 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:14.702328 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:14.702342 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:14.702443 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.702580 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.702661 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260619 06:52:14.703531 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.695066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:34085 (local address 127.14.116.193:34197)
0619 06:52:14.695895 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:14.695951 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:14.697350 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:14.697934 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:14.697941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:14.698020 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:14.698612 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.698633 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.699405 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.699410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.700407 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.700416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.701086 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.701089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.701314 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.702009 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:52:14.702010 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:14.702048 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:52:14.702105 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:14.702446 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:14.702453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:14.702456 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:52:14.702546 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:52:14.702745 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":146,"threads_started":1}
I20260619 06:52:14.704772 32594 heartbeater.cc:347] Connected to a master server at 127.14.116.254:34085
I20260619 06:52:14.705026 32594 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:14.705657 32594 heartbeater.cc:511] Master 127.14.116.254:34085 requested a full tablet report, sending...
I20260619 06:52:14.707206 32406 ts_manager.cc:194] Registered new tserver with Master: e9f51565fa0e4776a923b7e95d70829f (127.14.116.193:44123)
I20260619 06:52:14.708448 32406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.193:34197
W20260619 06:52:14.804632 32597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:14.804868 32597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:14.804905 32597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:14.808941 32597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:14.808992 32597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:14.809077 32597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:52:14.817081 32597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:14.818444 32597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:14.819880 32597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:14.829164 32603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:14.829233 32605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:14.829164 32602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:14.829319 32597 server_base.cc:1034] running on GCE node
I20260619 06:52:14.830041 32597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:14.830646 32597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:14.831800 32597 hybrid_clock.cc:648] HybridClock initialized: now 1781851934831785 us; error 25 us; skew 500 ppm
I20260619 06:52:14.833547 32597 webserver.cc:460] Webserver started at http://127.14.116.194:40193/ using document root <none> and password file <none>
I20260619 06:52:14.834122 32597 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:14.834184 32597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:14.834426 32597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:14.836453 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "ce3436c94ec146ccbcb3029ee38329c4"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "22f6fdf0ed4ffa7f22d9775829af3ce4"
server_key_iv: "5ec6d6953a03ef5c9c195abe749238d8"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.836995 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "ce3436c94ec146ccbcb3029ee38329c4"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "22f6fdf0ed4ffa7f22d9775829af3ce4"
server_key_iv: "5ec6d6953a03ef5c9c195abe749238d8"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.840477 32597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:52:14.842820 32611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.844102 32597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:52:14.844231 32597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "ce3436c94ec146ccbcb3029ee38329c4"
format_stamp: "Formatted at 2026-06-19 06:52:14 on dist-test-slave-9x3b"
server_key: "22f6fdf0ed4ffa7f22d9775829af3ce4"
server_key_iv: "5ec6d6953a03ef5c9c195abe749238d8"
server_key_version: "encryptionkey@0"
I20260619 06:52:14.844339 32597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:14.872880 32597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:14.873594 32597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:14.873804 32597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:14.874480 32597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:14.875408 32597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:14.875478 32597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.875561 32597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:14.875610 32597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:14.884840 32597 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:43881
I20260619 06:52:14.884872 32724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:43881 every 8 connection(s)
I20260619 06:52:14.885892 32597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:52:14.893497 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 32597
I20260619 06:52:14.893680 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:52:14.893981 14803 external_mini_cluster.cc:1428] Setting key 08dcd7dac765d05508f35d72038516ce
I20260619 06:52:14.895226 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.887736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:57601 (local address 127.14.116.254:34085)
0619 06:52:14.887883 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:14.887886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:14.888899 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 06:52:14.890293 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:14.890296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:14.890353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:14.890443 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:14.891316 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.891919 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.893076 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.893326 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.893437 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.894730 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:14.894733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:14.894736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:14.894750 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:14.894774 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:14.894782 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:14.894863 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:14.895028 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 06:52:14.895089 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:52:14.896160 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34085
--builtin_ntp_servers=127.14.116.212:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:52:14.896343 32727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:14.888151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:34085 (local address 127.14.116.194:57601)
0619 06:52:14.888755 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:14.888795 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:14.890084 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:14.890621 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:14.890628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:14.890710 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:14.891179 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.891191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.892050 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.892053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.892949 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:14.892958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:14.893482 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:14.893486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:14.893684 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:14.894422 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:52:14.894425 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:14.894481 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:52:14.894566 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:14.894866 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:14.894877 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:14.894886 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:52:14.895027 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:52:14.895308 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":114,"threads_started":1}
I20260619 06:52:14.897527 32725 heartbeater.cc:347] Connected to a master server at 127.14.116.254:34085
I20260619 06:52:14.897799 32725 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:14.898422 32725 heartbeater.cc:511] Master 127.14.116.254:34085 requested a full tablet report, sending...
I20260619 06:52:14.899535 32406 ts_manager.cc:194] Registered new tserver with Master: ce3436c94ec146ccbcb3029ee38329c4 (127.14.116.194:43881)
I20260619 06:52:14.900139 32406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.194:57601
W20260619 06:52:15.007179 32728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:15.007422 32728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:15.007483 32728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:15.011451 32728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:15.011507 32728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:15.011595 32728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:52:15.016386 32728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:15.017444 32728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:15.018764 32728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:15.027940 32734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:15.027933 32733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:15.028012 32728 server_base.cc:1034] running on GCE node
W20260619 06:52:15.027933 32736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:15.028550 32728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:15.029197 32728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:15.030421 32728 hybrid_clock.cc:648] HybridClock initialized: now 1781851935030404 us; error 23 us; skew 500 ppm
I20260619 06:52:15.032333 32728 webserver.cc:460] Webserver started at http://127.14.116.195:41267/ using document root <none> and password file <none>
I20260619 06:52:15.032840 32728 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:15.032896 32728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:15.033051 32728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:15.035025 32728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "6a945ca6cb56416eae2ec8a5a22784c5"
format_stamp: "Formatted at 2026-06-19 06:52:15 on dist-test-slave-9x3b"
server_key: "726d5b537da922ab2ee1ec9b3d23fec6"
server_key_iv: "d4bc314671fdd6132c101d3862829ab9"
server_key_version: "encryptionkey@0"
I20260619 06:52:15.035524 32728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "6a945ca6cb56416eae2ec8a5a22784c5"
format_stamp: "Formatted at 2026-06-19 06:52:15 on dist-test-slave-9x3b"
server_key: "726d5b537da922ab2ee1ec9b3d23fec6"
server_key_iv: "d4bc314671fdd6132c101d3862829ab9"
server_key_version: "encryptionkey@0"
I20260619 06:52:15.038954 32728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 06:52:15.041247 32742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:15.042443 32728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:52:15.042529 32728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "6a945ca6cb56416eae2ec8a5a22784c5"
format_stamp: "Formatted at 2026-06-19 06:52:15 on dist-test-slave-9x3b"
server_key: "726d5b537da922ab2ee1ec9b3d23fec6"
server_key_iv: "d4bc314671fdd6132c101d3862829ab9"
server_key_version: "encryptionkey@0"
I20260619 06:52:15.042613 32728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:15.054548 32728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:15.055074 32728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:15.055205 32728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:15.055748 32728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:15.056586 32728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:15.056631 32728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:15.056668 32728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:15.056700 32728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:15.066612 32728 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:34303
I20260619 06:52:15.066633 387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:34303 every 8 connection(s)
I20260619 06:52:15.067797 32728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:52:15.072708 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 32728
I20260619 06:52:15.072822 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:52:15.073097 14803 external_mini_cluster.cc:1428] Setting key 584771795783088104cbc6b11709d4ec
I20260619 06:52:15.077365 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.069850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:48941 (local address 127.14.116.254:34085)
0619 06:52:15.070007 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:15.070012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:15.070995 (+ 983us) server_negotiation.cc:407] Connection header received
0619 06:52:15.072366 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:15.072369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:15.072416 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:15.072523 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:15.073492 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.074262 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.075643 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.075877 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.075958 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.076998 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:15.077000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:15.077002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:15.077012 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:15.077027 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:15.077034 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:15.077087 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:15.077215 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 06:52:15.077259 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:52:15.078190 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.070233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:34085 (local address 127.14.116.195:48941)
0619 06:52:15.070850 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:15.070888 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:15.072177 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:15.072664 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:15.072670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:15.072749 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:15.073333 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.073348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.074403 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.074405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.075505 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.075515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.076001 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.076003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.076133 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.076781 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:52:15.076783 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:15.076820 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:52:15.076879 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:15.077102 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:15.077108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:15.077112 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:52:15.077202 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:52:15.077432 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260619 06:52:15.079321 388 heartbeater.cc:347] Connected to a master server at 127.14.116.254:34085
I20260619 06:52:15.079593 388 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:15.080224 388 heartbeater.cc:511] Master 127.14.116.254:34085 requested a full tablet report, sending...
I20260619 06:52:15.081250 32406 ts_manager.cc:194] Registered new tserver with Master: 6a945ca6cb56416eae2ec8a5a22784c5 (127.14.116.195:34303)
I20260619 06:52:15.081880 32406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.195:48941
I20260619 06:52:15.086606 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:52:15.092128 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.088365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.14.116.254:34085)
0619 06:52:15.088482 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:15.088507 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:15.088722 (+ 215us) server_negotiation.cc:407] Connection header received
0619 06:52:15.088872 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:15.088875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:15.088916 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:15.089007 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:15.089275 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.089764 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.090588 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.090818 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.090931 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.091688 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:15.091692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:15.091695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:15.091707 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:15.091734 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:15.091742 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:15.091817 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:15.091941 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 06:52:15.091994 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:52:15.094877 32406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:52:15.097777 32406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:52:15.111191 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.105515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:44123 (local address 127.0.0.1:47660)
0619 06:52:15.106479 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:15.106523 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:15.106654 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:15.107233 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:15.107240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:15.107268 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:15.107412 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.107420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.108910 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.108916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.109908 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.109917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.110679 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.110683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.110780 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.110816 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:52:15.110934 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 06:52:15.111046 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":47,"spinlock_wait_cycles":2688,"thread_start_us":83,"threads_started":1}
I20260619 06:52:15.111255 404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.106139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.14.116.193:44123)
0619 06:52:15.106536 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:15.106552 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:15.106581 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:52:15.106839 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:15.106846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:15.107055 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:15.107210 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:15.107540 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.108776 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.110032 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.110556 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.110688 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.110874 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:15.110998 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 06:52:15.111099 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":107,"threads_started":1}
I20260619 06:52:15.112102 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.105778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34303 (local address 127.0.0.1:33222)
0619 06:52:15.106698 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:15.106712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:15.106860 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:15.107482 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:15.107487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:15.107508 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:15.107649 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.107658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.108597 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.108603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.110523 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.110528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.111667 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.111673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.111754 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.111772 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 06:52:15.111895 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 06:52:15.111957 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":63,"thread_start_us":104,"threads_started":1}
I20260619 06:52:15.112531 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.106012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.14.116.195:34303)
0619 06:52:15.106328 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:15.106339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:15.106787 (+ 448us) server_negotiation.cc:407] Connection header received
0619 06:52:15.106991 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:15.106995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:15.107189 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:15.107340 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:15.107772 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.108487 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.110636 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.111312 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.111422 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.111580 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:15.112256 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 06:52:15.112365 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260619 06:52:15.112591 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.105363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:43881 (local address 127.0.0.1:45266)
0619 06:52:15.106472 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:15.106521 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:15.106654 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:15.107491 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:15.107496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:15.107516 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:15.107663 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.107670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.108986 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.108989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.109913 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.109922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.112057 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.112060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.112142 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.112335 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 06:52:15.112395 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 06:52:15.112469 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":68,"spinlock_wait_cycles":6400,"thread_start_us":87,"threads_started":1}
I20260619 06:52:15.113278 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.105512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.14.116.194:43881)
0619 06:52:15.106956 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:15.106967 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:15.106987 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:52:15.107054 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:15.107059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:15.107217 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:15.107343 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:15.107801 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.108881 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.110945 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.111925 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.112855 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.113003 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:15.113049 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 06:52:15.113130 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":112,"threads_started":1}
I20260619 06:52:15.114569 32528 tablet_service.cc:1467] Processing CreateTablet for tablet c148ed3d0ce74090957dd642b47848e0 (DEFAULT_TABLE table=test-table [id=fe10b3fdef174e4db02344c0bbeb1701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:15.115015 322 tablet_service.cc:1467] Processing CreateTablet for tablet c148ed3d0ce74090957dd642b47848e0 (DEFAULT_TABLE table=test-table [id=fe10b3fdef174e4db02344c0bbeb1701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:15.115511 32528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c148ed3d0ce74090957dd642b47848e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:15.116030 322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c148ed3d0ce74090957dd642b47848e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:15.115823 32659 tablet_service.cc:1467] Processing CreateTablet for tablet c148ed3d0ce74090957dd642b47848e0 (DEFAULT_TABLE table=test-table [id=fe10b3fdef174e4db02344c0bbeb1701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:15.116731 32659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c148ed3d0ce74090957dd642b47848e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:15.121418 405 tablet_bootstrap.cc:492] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f: Bootstrap starting.
I20260619 06:52:15.123155 407 tablet_bootstrap.cc:492] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: Bootstrap starting.
I20260619 06:52:15.123530 406 tablet_bootstrap.cc:492] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: Bootstrap starting.
I20260619 06:52:15.123742 405 tablet_bootstrap.cc:654] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:15.124604 405 log.cc:824] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:15.125223 407 tablet_bootstrap.cc:654] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:15.125394 406 tablet_bootstrap.cc:654] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:15.126056 407 log.cc:824] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:15.126134 406 log.cc:824] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:15.127147 405 tablet_bootstrap.cc:492] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f: No bootstrap required, opened a new log
I20260619 06:52:15.127360 405 ts_tablet_manager.cc:1397] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:52:15.128381 407 tablet_bootstrap.cc:492] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: No bootstrap required, opened a new log
I20260619 06:52:15.128408 406 tablet_bootstrap.cc:492] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: No bootstrap required, opened a new log
I20260619 06:52:15.128556 406 ts_tablet_manager.cc:1397] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:52:15.128571 407 ts_tablet_manager.cc:1397] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:52:15.130808 405 raft_consensus.cc:348] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } }
I20260619 06:52:15.131043 405 raft_consensus.cc:374] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:15.131100 405 raft_consensus.cc:729] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f51565fa0e4776a923b7e95d70829f, State: Initialized, Role: FOLLOWER
I20260619 06:52:15.131551 405 consensus_queue.cc:260] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } }
I20260619 06:52:15.132231 32594 heartbeater.cc:503] Master 127.14.116.254:34085 was elected leader, sending a full tablet report...
I20260619 06:52:15.132582 405 ts_tablet_manager.cc:1428] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:52:15.132416 407 raft_consensus.cc:348] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } }
I20260619 06:52:15.132478 406 raft_consensus.cc:348] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } }
I20260619 06:52:15.132652 407 raft_consensus.cc:374] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:15.132712 407 raft_consensus.cc:729] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3436c94ec146ccbcb3029ee38329c4, State: Initialized, Role: FOLLOWER
I20260619 06:52:15.132663 406 raft_consensus.cc:374] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:15.133015 406 raft_consensus.cc:729] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a945ca6cb56416eae2ec8a5a22784c5, State: Initialized, Role: FOLLOWER
I20260619 06:52:15.133134 407 consensus_queue.cc:260] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } }
I20260619 06:52:15.133460 406 consensus_queue.cc:260] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } }
I20260619 06:52:15.134001 407 ts_tablet_manager.cc:1428] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260619 06:52:15.134322 32725 heartbeater.cc:503] Master 127.14.116.254:34085 was elected leader, sending a full tablet report...
I20260619 06:52:15.134361 388 heartbeater.cc:503] Master 127.14.116.254:34085 was elected leader, sending a full tablet report...
I20260619 06:52:15.135102 406 ts_tablet_manager.cc:1428] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 06:52:15.136879 32726 tablet.cc:2369] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:52:15.179041 411 raft_consensus.cc:482] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:52:15.179239 411 raft_consensus.cc:504] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } }
I20260619 06:52:15.180624 411 leader_election.cc:288] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce3436c94ec146ccbcb3029ee38329c4 (127.14.116.194:43881), 6a945ca6cb56416eae2ec8a5a22784c5 (127.14.116.195:34303)
I20260619 06:52:15.184489 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.180948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:47669 (local address 127.14.116.194:43881)
0619 06:52:15.181067 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:15.181072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:15.181089 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:52:15.181147 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:15.181150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:15.181202 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:15.181289 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:15.181618 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.182241 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.183381 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.183889 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.183980 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.184016 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:15.184281 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 06:52:15.184367 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:52:15.184484 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.180829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:43881 (local address 127.14.116.193:47669)
0619 06:52:15.180950 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:15.180966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:15.181056 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:15.181311 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:15.181313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:15.181362 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:15.181487 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.181496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.182356 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.182359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.183252 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.183261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.184002 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.184006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.184124 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.184150 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:52:15.184235 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:52:15.184319 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":17}
I20260619 06:52:15.184584 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.180959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:45863 (local address 127.14.116.195:34303)
0619 06:52:15.181075 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:15.181080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:15.181367 (+ 287us) server_negotiation.cc:407] Connection header received
0619 06:52:15.181559 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:15.181566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:15.181620 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:15.181690 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:15.181936 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.182456 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.183523 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.184017 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.184135 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.184178 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:15.184393 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 06:52:15.184463 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:52:15.184804 414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.180829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34303 (local address 127.14.116.193:45863)
0619 06:52:15.181272 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:15.181287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:15.181387 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:15.181712 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:15.181714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:15.181732 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:15.181852 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.181859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.182560 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.182562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.183399 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:15.183408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.184125 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.184128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:15.184236 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.184280 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 06:52:15.184367 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:52:15.184430 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":145,"thread_start_us":110,"threads_started":1}
I20260619 06:52:15.185155 32679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c148ed3d0ce74090957dd642b47848e0" candidate_uuid: "e9f51565fa0e4776a923b7e95d70829f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3436c94ec146ccbcb3029ee38329c4" is_pre_election: true
I20260619 06:52:15.185454 32679 raft_consensus.cc:2454] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f51565fa0e4776a923b7e95d70829f in term 0.
I20260619 06:52:15.185509 330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c148ed3d0ce74090957dd642b47848e0" candidate_uuid: "e9f51565fa0e4776a923b7e95d70829f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" is_pre_election: true
I20260619 06:52:15.185798 330 raft_consensus.cc:2454] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f51565fa0e4776a923b7e95d70829f in term 0.
I20260619 06:52:15.186012 32482 leader_election.cc:302] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce3436c94ec146ccbcb3029ee38329c4, e9f51565fa0e4776a923b7e95d70829f; no voters:
I20260619 06:52:15.186276 411 raft_consensus.cc:2781] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:52:15.186359 411 raft_consensus.cc:482] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:52:15.186403 411 raft_consensus.cc:3037] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:15.187489 411 raft_consensus.cc:504] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } }
I20260619 06:52:15.187883 411 leader_election.cc:288] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [CANDIDATE]: Term 1 election: Requested vote from peers ce3436c94ec146ccbcb3029ee38329c4 (127.14.116.194:43881), 6a945ca6cb56416eae2ec8a5a22784c5 (127.14.116.195:34303)
I20260619 06:52:15.188158 32679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c148ed3d0ce74090957dd642b47848e0" candidate_uuid: "e9f51565fa0e4776a923b7e95d70829f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3436c94ec146ccbcb3029ee38329c4"
I20260619 06:52:15.188235 330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c148ed3d0ce74090957dd642b47848e0" candidate_uuid: "e9f51565fa0e4776a923b7e95d70829f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a945ca6cb56416eae2ec8a5a22784c5"
I20260619 06:52:15.188302 32679 raft_consensus.cc:3037] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:15.188369 330 raft_consensus.cc:3037] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:15.189298 32679 raft_consensus.cc:2454] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f51565fa0e4776a923b7e95d70829f in term 1.
I20260619 06:52:15.189428 330 raft_consensus.cc:2454] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f51565fa0e4776a923b7e95d70829f in term 1.
I20260619 06:52:15.189613 32482 leader_election.cc:302] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce3436c94ec146ccbcb3029ee38329c4, e9f51565fa0e4776a923b7e95d70829f; no voters:
I20260619 06:52:15.189785 411 raft_consensus.cc:2781] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:15.190025 411 raft_consensus.cc:686] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 1 LEADER]: Becoming Leader. State: Replica: e9f51565fa0e4776a923b7e95d70829f, State: Running, Role: LEADER
I20260619 06:52:15.190353 411 consensus_queue.cc:237] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } }
I20260619 06:52:15.193308 32405 catalog_manager.cc:5495] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f reported cstate change: term changed from 0 to 1, leader changed from <none> to e9f51565fa0e4776a923b7e95d70829f (127.14.116.193). New cstate: current_term: 1 leader_uuid: "e9f51565fa0e4776a923b7e95d70829f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9f51565fa0e4776a923b7e95d70829f" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 } health_report { overall_health: UNKNOWN } } }
W20260619 06:52:15.193872 32595 tablet.cc:2369] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:52:15.207078 404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.203504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.14.116.193:44123)
0619 06:52:15.203671 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:15.203675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:15.203689 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:52:15.203856 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:15.203859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:15.203911 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:15.204021 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:15.204254 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.204748 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.205617 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:15.205876 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:15.205987 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:15.206282 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:15.206672 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:15.206805 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:15.206874 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 06:52:15.206927 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:52:15.212569 32679 raft_consensus.cc:1261] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [term 1 FOLLOWER]: Refusing update from remote peer e9f51565fa0e4776a923b7e95d70829f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:15.212569 330 raft_consensus.cc:1261] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [term 1 FOLLOWER]: Refusing update from remote peer e9f51565fa0e4776a923b7e95d70829f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:15.213269 415 consensus_queue.cc:1035] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce3436c94ec146ccbcb3029ee38329c4" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 43881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:15.213476 411 consensus_queue.cc:1035] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a945ca6cb56416eae2ec8a5a22784c5" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:15.221602 419 mvcc.cc:205] Tried to move back new op lower bound from 7298465526623547392 to 7298465526542364672. Current Snapshot: MvccSnapshot[applied={T|T < 7298465526623547392}]
I20260619 06:52:15.222620 420 mvcc.cc:205] Tried to move back new op lower bound from 7298465526623547392 to 7298465526542364672. Current Snapshot: MvccSnapshot[applied={T|T < 7298465526623547392}]
I20260619 06:52:15.229307 32405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:15.229435 32405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:15.231724 32405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0de89668941420a91a2e185f974dee3: Sending DeleteTablet for 3 replicas of tablet c148ed3d0ce74090957dd642b47848e0
I20260619 06:52:15.232409 322 tablet_service.cc:1514] Processing DeleteTablet for tablet c148ed3d0ce74090957dd642b47848e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:15 UTC) from {username='slave'} at 127.0.0.1:33222
I20260619 06:52:15.232432 32528 tablet_service.cc:1514] Processing DeleteTablet for tablet c148ed3d0ce74090957dd642b47848e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:15 UTC) from {username='slave'} at 127.0.0.1:47660
I20260619 06:52:15.232841 427 tablet_replica.cc:330] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: stopping tablet replica
I20260619 06:52:15.232865 32659 tablet_service.cc:1514] Processing DeleteTablet for tablet c148ed3d0ce74090957dd642b47848e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:15 UTC) from {username='slave'} at 127.0.0.1:45266
I20260619 06:52:15.233179 427 raft_consensus.cc:2229] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:15.233389 428 tablet_replica.cc:330] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f: stopping tablet replica
I20260619 06:52:15.233554 427 raft_consensus.cc:2258] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:15.233708 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 32466
I20260619 06:52:15.233719 428 raft_consensus.cc:2229] T c148ed3d0ce74090957dd642b47848e0 P e9f51565fa0e4776a923b7e95d70829f [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:52:15.235466 427 ts_tablet_manager.cc:1905] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:15.235874 429 tablet_replica.cc:330] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: stopping tablet replica
I20260619 06:52:15.236123 429 raft_consensus.cc:2229] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:15.236390 429 raft_consensus.cc:2258] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:15.237444 429 ts_tablet_manager.cc:1905] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:15.239209 427 ts_tablet_manager.cc:1918] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:15.239310 427 log.cc:1196] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/c148ed3d0ce74090957dd642b47848e0
I20260619 06:52:15.239404 429 ts_tablet_manager.cc:1918] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:15.239519 429 log.cc:1196] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/c148ed3d0ce74090957dd642b47848e0
I20260619 06:52:15.239655 427 ts_tablet_manager.cc:1939] T c148ed3d0ce74090957dd642b47848e0 P 6a945ca6cb56416eae2ec8a5a22784c5: Deleting consensus metadata
I20260619 06:52:15.239858 429 ts_tablet_manager.cc:1939] T c148ed3d0ce74090957dd642b47848e0 P ce3436c94ec146ccbcb3029ee38329c4: Deleting consensus metadata
I20260619 06:52:15.240809 32392 catalog_manager.cc:4841] TS 6a945ca6cb56416eae2ec8a5a22784c5 (127.14.116.195:34303): tablet c148ed3d0ce74090957dd642b47848e0 (table test-table [id=fe10b3fdef174e4db02344c0bbeb1701]) successfully deleted
I20260619 06:52:15.240811 32391 catalog_manager.cc:4841] TS ce3436c94ec146ccbcb3029ee38329c4 (127.14.116.194:43881): tablet c148ed3d0ce74090957dd642b47848e0 (table test-table [id=fe10b3fdef174e4db02344c0bbeb1701]) successfully deleted
W20260619 06:52:15.241430 32393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:44123 (error 108)
I20260619 06:52:15.241645 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 32597
I20260619 06:52:15.243062 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:15.242736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:44123 (local address 127.0.0.1:47680)
0619 06:52:15.242883 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:15.242963 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 06:52:15.243769 32393 catalog_manager.cc:4584] TS e9f51565fa0e4776a923b7e95d70829f (127.14.116.193:44123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c148ed3d0ce74090957dd642b47848e0: Network error: Client connection negotiation failed: client connection to 127.14.116.193:44123: connect: Connection refused (error 111)
I20260619 06:52:15.247033 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 32728
I20260619 06:52:15.252955 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 32375
2026-06-19T06:52:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (997 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:15 dist-test-slave-9x3b krb5kdc[433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:15 dist-test-slave-9x3b krb5kdc[433](info): set up 2 sockets
Jun 19 06:52:15 dist-test-slave-9x3b krb5kdc[433](info): commencing operation
krb5kdc: starting...
W20260619 06:52:16.287389 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:16 dist-test-slave-9x3b krb5kdc[433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:16.399763 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:43905
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:43905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:16.505458 446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:16.505749 446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:16.505831 446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:16.509824 446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:16.509907 446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:16.509943 446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:16.510002 446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:16.510038 446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:16.515167 446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:43905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:43905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:16.516240 446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:16.517226 446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:16.525291 452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:16.525297 451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:16.525297 454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:16.525645 446 server_base.cc:1034] running on GCE node
I20260619 06:52:16.526163 446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:16.527179 446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:16.528371 446 hybrid_clock.cc:648] HybridClock initialized: now 1781851936528344 us; error 44 us; skew 500 ppm
Jun 19 06:52:16 dist-test-slave-9x3b krb5kdc[433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851936, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:16.534087 446 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:16.534945 446 webserver.cc:460] Webserver started at http://127.14.116.254:45963/ using document root <none> and password file <none>
I20260619 06:52:16.535435 446 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:16.535491 446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:16.535722 446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:16.537648 446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "28322dcd223447479963963252e8c10c"
format_stamp: "Formatted at 2026-06-19 06:52:16 on dist-test-slave-9x3b"
server_key: "b3b504128158e06fd1d5b90f0912283d"
server_key_iv: "e95a16130c3bc1d4c19a9e12e397aced"
server_key_version: "encryptionkey@0"
I20260619 06:52:16.538138 446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "28322dcd223447479963963252e8c10c"
format_stamp: "Formatted at 2026-06-19 06:52:16 on dist-test-slave-9x3b"
server_key: "b3b504128158e06fd1d5b90f0912283d"
server_key_iv: "e95a16130c3bc1d4c19a9e12e397aced"
server_key_version: "encryptionkey@0"
I20260619 06:52:16.541602 446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:16.543768 461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:16.544902 446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:16.544997 446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "28322dcd223447479963963252e8c10c"
format_stamp: "Formatted at 2026-06-19 06:52:16 on dist-test-slave-9x3b"
server_key: "b3b504128158e06fd1d5b90f0912283d"
server_key_iv: "e95a16130c3bc1d4c19a9e12e397aced"
server_key_version: "encryptionkey@0"
I20260619 06:52:16.545076 446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:16.559995 446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:16.562551 446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:16.562693 446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:16.569435 446 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:43905
I20260619 06:52:16.569458 513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:43905 every 8 connection(s)
I20260619 06:52:16.570323 446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:16.572868 514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:16.575232 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 446
I20260619 06:52:16.575382 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:16.575716 14803 external_mini_cluster.cc:1428] Setting key 999f2e38ab72ca45fbff932523380217
I20260619 06:52:16.578845 514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c: Bootstrap starting.
I20260619 06:52:16.580981 514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:16.581673 514 log.cc:824] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:16.583879 514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c: No bootstrap required, opened a new log
Jun 19 06:52:16 dist-test-slave-9x3b krb5kdc[433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:16.586717 514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28322dcd223447479963963252e8c10c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43905 } }
I20260619 06:52:16.586962 514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:16.587040 514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28322dcd223447479963963252e8c10c, State: Initialized, Role: FOLLOWER
I20260619 06:52:16.587446 514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28322dcd223447479963963252e8c10c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43905 } }
I20260619 06:52:16.587598 514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:16.587682 514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:16.587782 514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:16.588604 514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28322dcd223447479963963252e8c10c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43905 } }
I20260619 06:52:16.588935 514 leader_election.cc:302] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28322dcd223447479963963252e8c10c; no voters:
I20260619 06:52:16.589272 514 leader_election.cc:288] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:16.589390 519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:16.589612 519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [term 1 LEADER]: Becoming Leader. State: Replica: 28322dcd223447479963963252e8c10c, State: Running, Role: LEADER
I20260619 06:52:16.589949 519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28322dcd223447479963963252e8c10c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43905 } }
I20260619 06:52:16.590478 514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:16.590674 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:16.577089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.14.116.254:43905)
0619 06:52:16.577503 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:16.577524 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:16.577566 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:52:16.578281 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:16.578300 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:16.578610 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:16.578936 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:16.579743 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:16.580716 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:16.581628 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:16.582278 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:16.582448 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:16.585084 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:16.585089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:16.585101 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:16.585126 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:16.587499 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:16.588045 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:16.588053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:16.588063 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:16.588169 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:16.588461 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:16.588464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:16.588465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:16.588911 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:16.589114 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:16.589355 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 06:52:16.589803 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260619 06:52:16.591050 519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28322dcd223447479963963252e8c10c. Latest consensus state: current_term: 1 leader_uuid: "28322dcd223447479963963252e8c10c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28322dcd223447479963963252e8c10c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43905 } } }
I20260619 06:52:16.591169 519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:16.592142 521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28322dcd223447479963963252e8c10c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28322dcd223447479963963252e8c10c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 43905 } } }
I20260619 06:52:16.592254 521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c [sys.catalog]: This master's current role is: LEADER
W20260619 06:52:16.595165 534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:16.595263 534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:16.595342 525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:16.596086 525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:16.601322 525 catalog_manager.cc:1347] Generated new cluster ID: a929e672800b494caca93d391ec89fa4
I20260619 06:52:16.601392 525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:16.622049 525 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:16.623054 525 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:16.630312 525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c: Generated new TSK 0
I20260619 06:52:16.630985 525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:52:16.664278 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43905
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:16.782585 540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:16.782879 540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:16.782972 540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:16.786980 540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:16.787070 540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:16.787156 540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:52:16.792800 540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:16.793879 540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:16.794926 540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:16.804316 545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:16.804323 546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:16.804339 548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:16.804697 540 server_base.cc:1034] running on GCE node
I20260619 06:52:16.805073 540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:16.805691 540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:16.806879 540 hybrid_clock.cc:648] HybridClock initialized: now 1781851936806864 us; error 27 us; skew 500 ppm
Jun 19 06:52:16 dist-test-slave-9x3b krb5kdc[433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851936, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:16.809535 540 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:52:16.810329 540 webserver.cc:460] Webserver started at http://127.14.116.193:39555/ using document root <none> and password file <none>
I20260619 06:52:16.810832 540 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:16.810894 540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:16.811053 540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:16.813042 540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "678b83a45f544f3b8e7776203bcbbf9e"
format_stamp: "Formatted at 2026-06-19 06:52:16 on dist-test-slave-9x3b"
server_key: "6257119ae138e3239c7e840c4e4d786c"
server_key_iv: "2bfed0780a5a5cad62051bb044fc9cd6"
server_key_version: "encryptionkey@0"
I20260619 06:52:16.813541 540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "678b83a45f544f3b8e7776203bcbbf9e"
format_stamp: "Formatted at 2026-06-19 06:52:16 on dist-test-slave-9x3b"
server_key: "6257119ae138e3239c7e840c4e4d786c"
server_key_iv: "2bfed0780a5a5cad62051bb044fc9cd6"
server_key_version: "encryptionkey@0"
I20260619 06:52:16.817114 540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:52:16.819381 555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:16.820585 540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:16.820713 540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "678b83a45f544f3b8e7776203bcbbf9e"
format_stamp: "Formatted at 2026-06-19 06:52:16 on dist-test-slave-9x3b"
server_key: "6257119ae138e3239c7e840c4e4d786c"
server_key_iv: "2bfed0780a5a5cad62051bb044fc9cd6"
server_key_version: "encryptionkey@0"
I20260619 06:52:16.820815 540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:16.836552 540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:16.839207 540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:16.839407 540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:16.839999 540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:16.840786 540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:16.840830 540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:16.840906 540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:16.840938 540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:16.849648 540 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:37085
I20260619 06:52:16.849679 668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:37085 every 8 connection(s)
I20260619 06:52:16.850497 540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
Jun 19 06:52:16 dist-test-slave-9x3b krb5kdc[433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851936, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:16.860587 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 540
I20260619 06:52:16.860730 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:52:16.861066 14803 external_mini_cluster.cc:1428] Setting key 487d3bb0cb12c909b654ae2664675246
I20260619 06:52:16.863946 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:16.852161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:52351 (local address 127.14.116.254:43905)
0619 06:52:16.852302 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:16.852305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:16.853099 (+ 794us) server_negotiation.cc:407] Connection header received
0619 06:52:16.854478 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:16.854482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:16.854537 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:16.854649 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:16.855821 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:16.856361 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:16.857493 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:16.857726 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:16.857838 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:16.860275 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:16.860278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:16.860280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:16.860303 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:16.861813 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:16.862355 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:16.862358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:16.862360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:16.862450 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:16.862855 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:16.862857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:16.862858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:16.863021 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:16.863095 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:16.863734 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 06:52:16.863825 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:52:16.864964 671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:16.852536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:43905 (local address 127.14.116.193:52351)
0619 06:52:16.852950 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:16.852992 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:16.854234 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:16.854817 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:16.854823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:16.855154 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:16.855680 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:16.855697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:16.856488 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:16.856492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:16.857368 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:16.857377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:16.857875 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:16.857880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:16.858044 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:16.858629 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:16.858644 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:16.860095 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:16.861921 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:16.861925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:16.861936 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:16.862215 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:16.862558 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:16.862561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:16.862562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:16.862752 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:16.863128 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:16.863136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:16.863463 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 06:52:16.863689 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 06:52:16.864052 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260619 06:52:16.866153 669 heartbeater.cc:347] Connected to a master server at 127.14.116.254:43905
I20260619 06:52:16.866381 669 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:16.866927 669 heartbeater.cc:511] Master 127.14.116.254:43905 requested a full tablet report, sending...
I20260619 06:52:16.868568 478 ts_manager.cc:194] Registered new tserver with Master: 678b83a45f544f3b8e7776203bcbbf9e (127.14.116.193:37085)
I20260619 06:52:16.870173 478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:52351
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:52:16.886924 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43905
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:16.992244 674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:16.992524 674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:16.992595 674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:16.996564 674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:16.996616 674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:16.996721 674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:52:17.001873 674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:17.002812 674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:17.003775 674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:52:17.012681 674 server_base.cc:1034] running on GCE node
W20260619 06:52:17.012648 680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:17.012648 679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:17.012660 682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:17.013291 674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:17.013870 674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:17.015051 674 hybrid_clock.cc:648] HybridClock initialized: now 1781851937015038 us; error 22 us; skew 500 ppm
Jun 19 06:52:17 dist-test-slave-9x3b krb5kdc[433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:17.017663 674 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:52:17.018462 674 webserver.cc:460] Webserver started at http://127.14.116.194:36521/ using document root <none> and password file <none>
I20260619 06:52:17.019001 674 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:17.019088 674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:17.019305 674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:17.021534 674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "6bddda32aea6451f9213904221a0f174"
format_stamp: "Formatted at 2026-06-19 06:52:17 on dist-test-slave-9x3b"
server_key: "c848ad5143986f8814d4feb00abaf277"
server_key_iv: "655448d0cf2490ee86d6de9fc8f72c06"
server_key_version: "encryptionkey@0"
I20260619 06:52:17.022166 674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "6bddda32aea6451f9213904221a0f174"
format_stamp: "Formatted at 2026-06-19 06:52:17 on dist-test-slave-9x3b"
server_key: "c848ad5143986f8814d4feb00abaf277"
server_key_iv: "655448d0cf2490ee86d6de9fc8f72c06"
server_key_version: "encryptionkey@0"
I20260619 06:52:17.025740 674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:17.027987 689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:17.029168 674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:17.029297 674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "6bddda32aea6451f9213904221a0f174"
format_stamp: "Formatted at 2026-06-19 06:52:17 on dist-test-slave-9x3b"
server_key: "c848ad5143986f8814d4feb00abaf277"
server_key_iv: "655448d0cf2490ee86d6de9fc8f72c06"
server_key_version: "encryptionkey@0"
I20260619 06:52:17.029405 674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:17.062076 674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:17.066442 674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:17.066649 674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:17.067363 674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:17.068410 674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:17.068490 674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:17.068559 674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:17.068590 674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:17.077616 674 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:39965
I20260619 06:52:17.077637 802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:39965 every 8 connection(s)
I20260619 06:52:17.078481 674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:52:17.082834 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 674
I20260619 06:52:17.082971 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:52:17.083287 14803 external_mini_cluster.cc:1428] Setting key e262877b69b245a23efed49a2090d85d
Jun 19 06:52:17 dist-test-slave-9x3b krb5kdc[433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:52:17.094097 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.080295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:52677 (local address 127.14.116.254:43905)
0619 06:52:17.080438 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.080443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.081281 (+ 838us) server_negotiation.cc:407] Connection header received
0619 06:52:17.082561 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.082566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.082624 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.082702 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:17.084197 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.084908 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.086639 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.087036 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.087147 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.089863 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:17.089870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:17.089876 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:17.089910 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:17.091762 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:17.092404 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:17.092407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:17.092408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:17.092485 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:17.092933 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:17.092936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:17.092937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:17.093114 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:17.093192 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.093821 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.093954 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:52:17.094990 805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.080720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:43905 (local address 127.14.116.194:52677)
0619 06:52:17.081134 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:17.081175 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:17.082331 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:17.082923 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:17.082935 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:17.083405 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:17.084066 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.084081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.085034 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.085041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.086506 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.086519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.087180 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.087183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.087318 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.087973 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:17.087998 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:17.089668 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:17.091911 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:17.091917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:17.091933 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:17.092302 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:17.092586 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:17.092589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:17.092592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:17.092798 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:17.093223 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:17.093230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:17.093573 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 06:52:17.093784 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 06:52:17.094146 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260619 06:52:17.096108 803 heartbeater.cc:347] Connected to a master server at 127.14.116.254:43905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
I20260619 06:52:17.096370 803 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:17.096855 803 heartbeater.cc:511] Master 127.14.116.254:43905 requested a full tablet report, sending...
I20260619 06:52:17.097875 478 ts_manager.cc:194] Registered new tserver with Master: 6bddda32aea6451f9213904221a0f174 (127.14.116.194:39965)
I20260619 06:52:17.098430 478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:52677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:52:17.110132 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43905
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:33385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:17.216150 808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:17.216439 808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:17.216518 808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:17.220564 808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:17.220661 808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:17.220753 808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:52:17.226199 808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:33385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:43905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:17.227288 808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:17.228308 808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:17.237686 813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:17.237711 814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:17.237711 816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:17.238054 808 server_base.cc:1034] running on GCE node
I20260619 06:52:17.238416 808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:17.238992 808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:17.240175 808 hybrid_clock.cc:648] HybridClock initialized: now 1781851937240153 us; error 32 us; skew 500 ppm
Jun 19 06:52:17 dist-test-slave-9x3b krb5kdc[433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:17.242661 808 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:52:17.243515 808 webserver.cc:460] Webserver started at http://127.14.116.195:36943/ using document root <none> and password file <none>
I20260619 06:52:17.244092 808 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:17.244184 808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:17.244412 808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:17.246340 808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "6f731f7b14a54919b506a7ba8014150a"
format_stamp: "Formatted at 2026-06-19 06:52:17 on dist-test-slave-9x3b"
server_key: "e2d48ed2acf297ef19f5168d60972f27"
server_key_iv: "db4dfbe9490fa45c6926c1ed7de8760f"
server_key_version: "encryptionkey@0"
I20260619 06:52:17.246886 808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "6f731f7b14a54919b506a7ba8014150a"
format_stamp: "Formatted at 2026-06-19 06:52:17 on dist-test-slave-9x3b"
server_key: "e2d48ed2acf297ef19f5168d60972f27"
server_key_iv: "db4dfbe9490fa45c6926c1ed7de8760f"
server_key_version: "encryptionkey@0"
I20260619 06:52:17.250440 808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:52:17.252743 823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:17.253921 808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:52:17.254053 808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "6f731f7b14a54919b506a7ba8014150a"
format_stamp: "Formatted at 2026-06-19 06:52:17 on dist-test-slave-9x3b"
server_key: "e2d48ed2acf297ef19f5168d60972f27"
server_key_iv: "db4dfbe9490fa45c6926c1ed7de8760f"
server_key_version: "encryptionkey@0"
I20260619 06:52:17.254159 808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:17.268007 808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:17.270819 808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:17.271015 808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:17.271574 808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:17.272415 808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:17.272483 808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:17.272553 808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:17.272605 808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:17.281025 808 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:34499
I20260619 06:52:17.281050 936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:34499 every 8 connection(s)
I20260619 06:52:17.281841 808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:52:17.285766 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 808
I20260619 06:52:17.285917 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:52:17.286192 14803 external_mini_cluster.cc:1428] Setting key c8fea4f886d8bdc533df3ca74abd050d
Jun 19 06:52:17 dist-test-slave-9x3b krb5kdc[433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851937, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:17.295089 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.283512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:59533 (local address 127.14.116.254:43905)
0619 06:52:17.283701 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.283706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.284487 (+ 781us) server_negotiation.cc:407] Connection header received
0619 06:52:17.285730 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.285735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.285794 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.285897 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:17.287240 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.287968 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.289148 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.289361 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.289441 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.292068 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:17.292070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:17.292072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:17.292095 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:17.293378 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:17.293825 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:17.293829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:17.293830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:17.293898 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:17.294180 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:17.294182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:17.294183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:17.294362 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:17.294435 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.294883 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.294975 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:52:17.295924 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.283922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:43905 (local address 127.14.116.195:59533)
0619 06:52:17.284328 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:17.284382 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:17.285535 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:17.286090 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:17.286096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:17.286511 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:17.287091 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.287112 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.288098 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.288101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.289045 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.289054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.289507 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.289511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.289654 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.290270 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:17.290289 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:17.291928 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:17.293479 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:17.293484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:17.293495 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:17.293734 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:17.293975 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:17.293977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:17.293979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:17.294104 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:17.294452 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:17.294457 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:17.294674 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 06:52:17.294839 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 06:52:17.295072 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260619 06:52:17.297042 937 heartbeater.cc:347] Connected to a master server at 127.14.116.254:43905
I20260619 06:52:17.297276 937 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:17.297735 937 heartbeater.cc:511] Master 127.14.116.254:43905 requested a full tablet report, sending...
I20260619 06:52:17.298707 478 ts_manager.cc:194] Registered new tserver with Master: 6f731f7b14a54919b506a7ba8014150a (127.14.116.195:34499)
I20260619 06:52:17.299252 478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:59533
I20260619 06:52:17.300001 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:52:17.308925 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.302058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.14.116.254:43905)
0619 06:52:17.302178 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.302181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.302412 (+ 231us) server_negotiation.cc:407] Connection header received
0619 06:52:17.302488 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.302490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.302523 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.302587 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:17.303207 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.303769 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.304548 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.304752 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.304831 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.306225 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:17.306228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:17.306229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:17.306250 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:17.307424 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:17.307933 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:17.307936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:17.307937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:17.308004 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:17.308253 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:17.308255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:17.308256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:17.308438 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:17.308547 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.308747 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.308817 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:52:17.311525 478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:52:17.314462 478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:52:17.328600 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.322520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34499 (local address 127.0.0.1:37286)
0619 06:52:17.323333 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:17.323380 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:17.323500 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:17.324245 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:17.324247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:17.324272 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:17.324423 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.324430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.325667 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.325673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.327061 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.327072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.328072 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.328075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.328178 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.328212 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:52:17.328320 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:52:17.328434 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":87,"spinlock_wait_cycles":15232,"thread_start_us":95,"threads_started":1}
I20260619 06:52:17.328724 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.322455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:37085 (local address 127.0.0.1:59882)
0619 06:52:17.323336 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:17.323380 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:17.323529 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:17.324231 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:17.324237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:17.324268 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:17.324549 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.324554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.326013 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.326016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.327236 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.327243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.328324 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.328328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.328404 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.328447 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 06:52:17.328505 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 06:52:17.328559 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":116,"spinlock_wait_cycles":384,"thread_start_us":60,"threads_started":1}
I20260619 06:52:17.328795 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.322771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.14.116.193:37085)
0619 06:52:17.323365 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.323380 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.323470 (+ 90us) server_negotiation.cc:407] Connection header received
0619 06:52:17.323703 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.323710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.323912 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.324035 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:17.324663 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.325881 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.327350 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.328208 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.328361 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.328487 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.328537 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.328639 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":117,"threads_started":1}
I20260619 06:52:17.329149 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.323175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.14.116.195:34499)
0619 06:52:17.323800 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.323810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.323841 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:52:17.323908 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.323912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.324068 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.324410 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:17.324552 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.325542 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.327182 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.327951 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.328086 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.328213 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.328950 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.329047 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":100,"threads_started":1}
I20260619 06:52:17.329247 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.322765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:39965 (local address 127.0.0.1:54834)
0619 06:52:17.323502 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:17.323516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:17.323603 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:17.324727 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:17.324730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:17.324747 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:17.324831 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.324836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.326168 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.326173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.327333 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.327343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.328403 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.328407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.328525 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.328550 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:52:17.328610 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 06:52:17.329129 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":24,"thread_start_us":94,"threads_started":1}
I20260619 06:52:17.329852 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.323521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.14.116.194:39965)
0619 06:52:17.324256 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.324279 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.324301 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:52:17.324376 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.324381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.324531 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.324733 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:17.324948 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.326040 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.327454 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.328279 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.329478 (+ 1199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.329600 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.329646 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.329734 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":168,"threads_started":1}
I20260619 06:52:17.331585 871 tablet_service.cc:1467] Processing CreateTablet for tablet 165461485e064be68fcbe9f5758c6e50 (DEFAULT_TABLE table=test-table [id=51858caa364d41078b704eb78acfb8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:17.331666 603 tablet_service.cc:1467] Processing CreateTablet for tablet 165461485e064be68fcbe9f5758c6e50 (DEFAULT_TABLE table=test-table [id=51858caa364d41078b704eb78acfb8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:17.332559 871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165461485e064be68fcbe9f5758c6e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:17.332286 737 tablet_service.cc:1467] Processing CreateTablet for tablet 165461485e064be68fcbe9f5758c6e50 (DEFAULT_TABLE table=test-table [id=51858caa364d41078b704eb78acfb8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:17.332716 603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165461485e064be68fcbe9f5758c6e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:17.333184 737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 165461485e064be68fcbe9f5758c6e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:17.338582 954 tablet_bootstrap.cc:492] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: Bootstrap starting.
I20260619 06:52:17.340154 956 tablet_bootstrap.cc:492] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: Bootstrap starting.
I20260619 06:52:17.340880 954 tablet_bootstrap.cc:654] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:17.342192 954 log.cc:824] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:17.342298 956 tablet_bootstrap.cc:654] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:17.342960 955 tablet_bootstrap.cc:492] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e: Bootstrap starting.
I20260619 06:52:17.343511 956 log.cc:824] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:17.344132 954 tablet_bootstrap.cc:492] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: No bootstrap required, opened a new log
I20260619 06:52:17.344336 954 ts_tablet_manager.cc:1397] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:52:17.344995 955 tablet_bootstrap.cc:654] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:17.345559 956 tablet_bootstrap.cc:492] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: No bootstrap required, opened a new log
I20260619 06:52:17.345736 956 ts_tablet_manager.cc:1397] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 06:52:17.346196 955 log.cc:824] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:17.347641 954 raft_consensus.cc:348] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } }
I20260619 06:52:17.347925 954 raft_consensus.cc:374] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:17.347991 954 raft_consensus.cc:729] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f731f7b14a54919b506a7ba8014150a, State: Initialized, Role: FOLLOWER
I20260619 06:52:17.348281 955 tablet_bootstrap.cc:492] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e: No bootstrap required, opened a new log
I20260619 06:52:17.348453 955 ts_tablet_manager.cc:1397] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:52:17.348474 954 consensus_queue.cc:260] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } }
I20260619 06:52:17.349112 937 heartbeater.cc:503] Master 127.14.116.254:43905 was elected leader, sending a full tablet report...
I20260619 06:52:17.349450 954 ts_tablet_manager.cc:1428] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:52:17.349427 956 raft_consensus.cc:348] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } }
I20260619 06:52:17.349650 956 raft_consensus.cc:374] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:17.349701 956 raft_consensus.cc:729] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bddda32aea6451f9213904221a0f174, State: Initialized, Role: FOLLOWER
I20260619 06:52:17.350075 956 consensus_queue.cc:260] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } }
I20260619 06:52:17.350683 803 heartbeater.cc:503] Master 127.14.116.254:43905 was elected leader, sending a full tablet report...
I20260619 06:52:17.350971 956 ts_tablet_manager.cc:1428] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 06:52:17.352085 955 raft_consensus.cc:348] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } }
I20260619 06:52:17.352288 955 raft_consensus.cc:374] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:17.352348 955 raft_consensus.cc:729] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678b83a45f544f3b8e7776203bcbbf9e, State: Initialized, Role: FOLLOWER
I20260619 06:52:17.352720 955 consensus_queue.cc:260] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } }
I20260619 06:52:17.353338 669 heartbeater.cc:503] Master 127.14.116.254:43905 was elected leader, sending a full tablet report...
I20260619 06:52:17.354765 955 ts_tablet_manager.cc:1428] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
W20260619 06:52:17.532727 938 tablet.cc:2369] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:52:17.567332 962 raft_consensus.cc:482] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:52:17.567549 962 raft_consensus.cc:504] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } }
I20260619 06:52:17.568665 962 leader_election.cc:288] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f731f7b14a54919b506a7ba8014150a (127.14.116.195:34499), 6bddda32aea6451f9213904221a0f174 (127.14.116.194:39965)
I20260619 06:52:17.573531 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.569124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:35553 (local address 127.14.116.194:39965)
0619 06:52:17.569262 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.569266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.569394 (+ 128us) server_negotiation.cc:407] Connection header received
0619 06:52:17.569555 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.569559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.569611 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.569695 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:17.569963 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.570710 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.572275 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.572898 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.572985 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.573036 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.573334 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.573408 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:52:17.573552 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.568919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:39965 (local address 127.14.116.193:35553)
0619 06:52:17.569285 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:17.569301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:17.569449 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:17.569708 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:17.569711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:17.569728 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:17.569846 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.569860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.570830 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.570833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.572142 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.572152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.573017 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.573020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.573125 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.573148 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:52:17.573321 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 06:52:17.573400 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":86,"threads_started":1}
I20260619 06:52:17.573777 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.569042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:49925 (local address 127.14.116.195:34499)
0619 06:52:17.569188 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.569195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.569540 (+ 345us) server_negotiation.cc:407] Connection header received
0619 06:52:17.569734 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.569737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.569786 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.569834 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:17.570224 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.570914 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.572152 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.572878 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.573008 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.573057 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.573505 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.573595 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:52:17.574162 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.568919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:34499 (local address 127.14.116.193:49925)
0619 06:52:17.569450 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:17.569463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:17.569613 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:17.569979 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:17.569981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:17.569995 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:17.570100 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.570106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.571023 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.571029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.572010 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:17.572023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.573122 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.573126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:17.573219 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.573241 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:52:17.573924 (+ 683us) client_negotiation.cc:240] Negotiation successful
0619 06:52:17.574006 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":117,"thread_start_us":90,"threads_started":1}
I20260619 06:52:17.574188 757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165461485e064be68fcbe9f5758c6e50" candidate_uuid: "678b83a45f544f3b8e7776203bcbbf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bddda32aea6451f9213904221a0f174" is_pre_election: true
I20260619 06:52:17.574555 757 raft_consensus.cc:2454] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 678b83a45f544f3b8e7776203bcbbf9e in term 0.
I20260619 06:52:17.574911 891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165461485e064be68fcbe9f5758c6e50" candidate_uuid: "678b83a45f544f3b8e7776203bcbbf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f731f7b14a54919b506a7ba8014150a" is_pre_election: true
I20260619 06:52:17.575059 559 leader_election.cc:302] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678b83a45f544f3b8e7776203bcbbf9e, 6bddda32aea6451f9213904221a0f174; no voters:
I20260619 06:52:17.575192 891 raft_consensus.cc:2454] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 678b83a45f544f3b8e7776203bcbbf9e in term 0.
I20260619 06:52:17.575340 962 raft_consensus.cc:2781] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:52:17.575419 962 raft_consensus.cc:482] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:52:17.575495 962 raft_consensus.cc:3037] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:17.576632 962 raft_consensus.cc:504] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } }
I20260619 06:52:17.577000 962 leader_election.cc:288] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [CANDIDATE]: Term 1 election: Requested vote from peers 6f731f7b14a54919b506a7ba8014150a (127.14.116.195:34499), 6bddda32aea6451f9213904221a0f174 (127.14.116.194:39965)
I20260619 06:52:17.577318 891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165461485e064be68fcbe9f5758c6e50" candidate_uuid: "678b83a45f544f3b8e7776203bcbbf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f731f7b14a54919b506a7ba8014150a"
I20260619 06:52:17.577344 757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "165461485e064be68fcbe9f5758c6e50" candidate_uuid: "678b83a45f544f3b8e7776203bcbbf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bddda32aea6451f9213904221a0f174"
I20260619 06:52:17.577454 891 raft_consensus.cc:3037] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:17.577463 757 raft_consensus.cc:3037] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:17.578457 891 raft_consensus.cc:2454] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678b83a45f544f3b8e7776203bcbbf9e in term 1.
I20260619 06:52:17.578755 557 leader_election.cc:302] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678b83a45f544f3b8e7776203bcbbf9e, 6f731f7b14a54919b506a7ba8014150a; no voters:
I20260619 06:52:17.578788 757 raft_consensus.cc:2454] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 678b83a45f544f3b8e7776203bcbbf9e in term 1.
I20260619 06:52:17.578953 962 raft_consensus.cc:2781] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:17.579181 962 raft_consensus.cc:686] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [term 1 LEADER]: Becoming Leader. State: Replica: 678b83a45f544f3b8e7776203bcbbf9e, State: Running, Role: LEADER
I20260619 06:52:17.579504 962 consensus_queue.cc:237] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } }
W20260619 06:52:17.579907 804 tablet.cc:2369] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:52:17.582937 477 catalog_manager.cc:5495] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 678b83a45f544f3b8e7776203bcbbf9e (127.14.116.193). New cstate: current_term: 1 leader_uuid: "678b83a45f544f3b8e7776203bcbbf9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "678b83a45f544f3b8e7776203bcbbf9e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 } health_report { overall_health: UNKNOWN } } }
W20260619 06:52:17.601680 670 tablet.cc:2369] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:52:17.622696 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.619202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.14.116.193:37085)
0619 06:52:17.619344 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.619358 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.619376 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:52:17.619463 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.619466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.619518 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.619643 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:17.619899 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.620403 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.621252 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.621481 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.621588 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.621885 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:17.622267 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:17.622433 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.622517 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.622555 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:52:17.628306 757 raft_consensus.cc:1261] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [term 1 FOLLOWER]: Refusing update from remote peer 678b83a45f544f3b8e7776203bcbbf9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:17.628319 891 raft_consensus.cc:1261] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [term 1 FOLLOWER]: Refusing update from remote peer 678b83a45f544f3b8e7776203bcbbf9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:17.628988 962 consensus_queue.cc:1035] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f731f7b14a54919b506a7ba8014150a" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 34499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:17.629185 965 consensus_queue.cc:1035] T 165461485e064be68fcbe9f5758c6e50 P 678b83a45f544f3b8e7776203bcbbf9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bddda32aea6451f9213904221a0f174" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:17.643836 477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:17.644022 477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:17.646229 477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28322dcd223447479963963252e8c10c: Sending DeleteTablet for 3 replicas of tablet 165461485e064be68fcbe9f5758c6e50
I20260619 06:52:17.646941 871 tablet_service.cc:1514] Processing DeleteTablet for tablet 165461485e064be68fcbe9f5758c6e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:17 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:37286
I20260619 06:52:17.646970 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 165461485e064be68fcbe9f5758c6e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:17 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:54834
I20260619 06:52:17.647420 978 tablet_replica.cc:330] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: stopping tablet replica
I20260619 06:52:17.647415 603 tablet_service.cc:1514] Processing DeleteTablet for tablet 165461485e064be68fcbe9f5758c6e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:17 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:59882
I20260619 06:52:17.647778 978 raft_consensus.cc:2229] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:17.647809 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 540
I20260619 06:52:17.648574 978 raft_consensus.cc:2258] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:17.649786 979 tablet_replica.cc:330] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: stopping tablet replica
I20260619 06:52:17.650110 978 ts_tablet_manager.cc:1905] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:17.650125 979 raft_consensus.cc:2229] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:17.650396 979 raft_consensus.cc:2258] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:17.651793 979 ts_tablet_manager.cc:1905] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:17.652406 978 ts_tablet_manager.cc:1918] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:17.652513 978 log.cc:1196] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/165461485e064be68fcbe9f5758c6e50
I20260619 06:52:17.652823 978 ts_tablet_manager.cc:1939] T 165461485e064be68fcbe9f5758c6e50 P 6f731f7b14a54919b506a7ba8014150a: Deleting consensus metadata
I20260619 06:52:17.653825 463 catalog_manager.cc:4841] TS 6f731f7b14a54919b506a7ba8014150a (127.14.116.195:34499): tablet 165461485e064be68fcbe9f5758c6e50 (table test-table [id=51858caa364d41078b704eb78acfb8a1]) successfully deleted
I20260619 06:52:17.655052 979 ts_tablet_manager.cc:1918] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:17.655148 979 log.cc:1196] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/165461485e064be68fcbe9f5758c6e50
I20260619 06:52:17.655452 979 ts_tablet_manager.cc:1939] T 165461485e064be68fcbe9f5758c6e50 P 6bddda32aea6451f9213904221a0f174: Deleting consensus metadata
W20260619 06:52:17.655709 462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:37085 (error 108)
I20260619 06:52:17.656477 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 674
I20260619 06:52:17.656629 465 catalog_manager.cc:4841] TS 6bddda32aea6451f9213904221a0f174 (127.14.116.194:39965): tablet 165461485e064be68fcbe9f5758c6e50 (table test-table [id=51858caa364d41078b704eb78acfb8a1]) successfully deleted
I20260619 06:52:17.656796 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.656433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:37085 (local address 127.0.0.1:59904)
0619 06:52:17.656626 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:17.656700 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260619 06:52:17.657783 462 catalog_manager.cc:4584] TS 678b83a45f544f3b8e7776203bcbbf9e (127.14.116.193:37085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 165461485e064be68fcbe9f5758c6e50: Network error: Client connection negotiation failed: client connection to 127.14.116.193:37085: connect: Connection refused (error 111)
I20260619 06:52:17.661795 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 808
I20260619 06:52:17.666990 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 446
2026-06-19T06:52:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2415 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T06:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:17Z Disabled control of system clock
I20260619 06:52:17.696995 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46683
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:46683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:52:17.806841 985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:17.807083 985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:17.807117 985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:17.811040 985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:17.811093 985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:17.811110 985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:17.811124 985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:17.811139 985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:17.815816 985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:46683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:17.816735 985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:17.817974 985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:17.825909 990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:17.825889 993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:17.825892 991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:17.826085 985 server_base.cc:1034] running on GCE node
I20260619 06:52:17.826764 985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:17.827792 985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:17.828974 985 hybrid_clock.cc:648] HybridClock initialized: now 1781851937828949 us; error 44 us; skew 500 ppm
I20260619 06:52:17.830700 985 webserver.cc:460] Webserver started at http://127.14.116.254:33409/ using document root <none> and password file <none>
I20260619 06:52:17.831254 985 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:17.831343 985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:17.831564 985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:17.833531 985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "b5bad7255f504fc995beef63f5848e20"
format_stamp: "Formatted at 2026-06-19 06:52:17 on dist-test-slave-9x3b"
server_key: "5758d697c094d9a51e5183b85b175f02"
server_key_iv: "33a2e326af493d009cd4f7af166e34d5"
server_key_version: "encryptionkey@0"
I20260619 06:52:17.834075 985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "b5bad7255f504fc995beef63f5848e20"
format_stamp: "Formatted at 2026-06-19 06:52:17 on dist-test-slave-9x3b"
server_key: "5758d697c094d9a51e5183b85b175f02"
server_key_iv: "33a2e326af493d009cd4f7af166e34d5"
server_key_version: "encryptionkey@0"
I20260619 06:52:17.837481 985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:17.839675 999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:17.840819 985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:17.840960 985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "b5bad7255f504fc995beef63f5848e20"
format_stamp: "Formatted at 2026-06-19 06:52:17 on dist-test-slave-9x3b"
server_key: "5758d697c094d9a51e5183b85b175f02"
server_key_iv: "33a2e326af493d009cd4f7af166e34d5"
server_key_version: "encryptionkey@0"
I20260619 06:52:17.841064 985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:17.862341 985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:17.862959 985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:17.863142 985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:17.869899 985 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:46683
I20260619 06:52:17.869912 1051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:46683 every 8 connection(s)
I20260619 06:52:17.870748 985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:17.872345 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 985
I20260619 06:52:17.872490 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:17.872776 14803 external_mini_cluster.cc:1428] Setting key 7d72fcbdeabef38f347ba992713d7528
I20260619 06:52:17.874099 1052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:17.879261 1052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20: Bootstrap starting.
I20260619 06:52:17.881433 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:17.874045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.14.116.254:46683)
0619 06:52:17.874654 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:17.874667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:17.874700 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:52:17.875336 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:17.875352 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:17.875789 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:17.876101 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:17.876494 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.877419 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.878275 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:17.878725 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:17.878904 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:17.879786 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:17.879791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:17.879801 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:17.880217 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:17.880284 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:17.880313 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:17.880427 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:17.880553 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 06:52:17.880800 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":151,"threads_started":1}
I20260619 06:52:17.881474 1052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:17.882191 1052 log.cc:824] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:17.884084 1052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20: No bootstrap required, opened a new log
I20260619 06:52:17.887266 1052 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bad7255f504fc995beef63f5848e20" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46683 } }
I20260619 06:52:17.887486 1052 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:17.887568 1052 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bad7255f504fc995beef63f5848e20, State: Initialized, Role: FOLLOWER
I20260619 06:52:17.887971 1052 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bad7255f504fc995beef63f5848e20" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46683 } }
I20260619 06:52:17.888124 1052 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:17.888209 1052 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:17.888311 1052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:17.889250 1052 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bad7255f504fc995beef63f5848e20" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46683 } }
I20260619 06:52:17.889587 1052 leader_election.cc:302] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5bad7255f504fc995beef63f5848e20; no voters:
I20260619 06:52:17.889869 1052 leader_election.cc:288] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:17.889963 1057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:17.890138 1057 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [term 1 LEADER]: Becoming Leader. State: Replica: b5bad7255f504fc995beef63f5848e20, State: Running, Role: LEADER
I20260619 06:52:17.890419 1057 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bad7255f504fc995beef63f5848e20" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46683 } }
I20260619 06:52:17.891044 1052 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:17.892073 1058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5bad7255f504fc995beef63f5848e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bad7255f504fc995beef63f5848e20" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46683 } } }
I20260619 06:52:17.892202 1058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:17.892112 1059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5bad7255f504fc995beef63f5848e20. Latest consensus state: current_term: 1 leader_uuid: "b5bad7255f504fc995beef63f5848e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5bad7255f504fc995beef63f5848e20" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 46683 } } }
I20260619 06:52:17.892402 1059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20 [sys.catalog]: This master's current role is: LEADER
W20260619 06:52:17.896085 1072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:17.896179 1072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:17.896255 1066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:17.896944 1066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:17.902050 1066 catalog_manager.cc:1347] Generated new cluster ID: 7c43b28b67cd45ee826fdf494970fffa
I20260619 06:52:17.902115 1066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:17.912088 1066 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:17.913126 1066 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:17.916224 1066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20: Generated new TSK 0
I20260619 06:52:17.916817 1066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:17.922629 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46683
--builtin_ntp_servers=127.14.116.212:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:52:18.035399 1076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:18.035650 1076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:18.035687 1076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:18.039737 1076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:18.039784 1076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:18.039877 1076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:52:18.044525 1076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:18.045501 1076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:18.046823 1076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:18.056087 1084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:18.056082 1082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:18.056082 1081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:18.056186 1076 server_base.cc:1034] running on GCE node
I20260619 06:52:18.056865 1076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:18.057466 1076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:18.058660 1076 hybrid_clock.cc:648] HybridClock initialized: now 1781851938058648 us; error 27 us; skew 500 ppm
I20260619 06:52:18.060542 1076 webserver.cc:460] Webserver started at http://127.14.116.193:40533/ using document root <none> and password file <none>
I20260619 06:52:18.061149 1076 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:18.061241 1076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:18.061480 1076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:18.063431 1076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "fd6bdbc761f64bbd8cec652da6052e0e"
format_stamp: "Formatted at 2026-06-19 06:52:18 on dist-test-slave-9x3b"
server_key: "0f5cd63683b4d6eb4a0e5f4a4231257b"
server_key_iv: "828ce9fc4501f3b5095a776195d5897d"
server_key_version: "encryptionkey@0"
I20260619 06:52:18.064046 1076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "fd6bdbc761f64bbd8cec652da6052e0e"
format_stamp: "Formatted at 2026-06-19 06:52:18 on dist-test-slave-9x3b"
server_key: "0f5cd63683b4d6eb4a0e5f4a4231257b"
server_key_iv: "828ce9fc4501f3b5095a776195d5897d"
server_key_version: "encryptionkey@0"
I20260619 06:52:18.067454 1076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:52:18.069878 1090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:18.071038 1076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:18.071175 1076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "fd6bdbc761f64bbd8cec652da6052e0e"
format_stamp: "Formatted at 2026-06-19 06:52:18 on dist-test-slave-9x3b"
server_key: "0f5cd63683b4d6eb4a0e5f4a4231257b"
server_key_iv: "828ce9fc4501f3b5095a776195d5897d"
server_key_version: "encryptionkey@0"
I20260619 06:52:18.071296 1076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:18.087092 1076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:18.087776 1076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:18.087965 1076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:18.088568 1076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:18.089478 1076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:18.089547 1076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:18.089620 1076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:18.089672 1076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:18.098438 1076 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:33655
I20260619 06:52:18.098474 1203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:33655 every 8 connection(s)
I20260619 06:52:18.099313 1076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:52:18.108222 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.101096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:60063 (local address 127.14.116.254:46683)
0619 06:52:18.101290 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.101295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.102301 (+ 1006us) server_negotiation.cc:407] Connection header received
0619 06:52:18.103662 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.103669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.103735 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.103821 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:18.104682 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.105232 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.106374 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.106607 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.106706 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.107783 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:18.107786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:18.107788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:18.107801 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:18.107822 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:18.107830 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:18.107901 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.108029 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.108102 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:52:18.109006 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 1076
I20260619 06:52:18.109148 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:52:18.109135 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.101528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46683 (local address 127.14.116.193:60063)
0619 06:52:18.102164 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:18.102203 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:18.103417 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:18.103976 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:18.103983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:18.104062 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:18.104542 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.104553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.105357 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.105360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.106241 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.106250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.106758 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.106761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.106914 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.107516 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:52:18.107518 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:18.107556 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:52:18.107638 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:18.107910 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:18.107916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:18.107920 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:52:18.108012 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:52:18.108227 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260619 06:52:18.109520 14803 external_mini_cluster.cc:1428] Setting key 2576fc1ca99efcc160247560681b0f51
I20260619 06:52:18.110421 1204 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46683
I20260619 06:52:18.110638 1204 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:18.111177 1204 heartbeater.cc:511] Master 127.14.116.254:46683 requested a full tablet report, sending...
I20260619 06:52:18.111405 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46683
--builtin_ntp_servers=127.14.116.212:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:52:18.112659 1016 ts_manager.cc:194] Registered new tserver with Master: fd6bdbc761f64bbd8cec652da6052e0e (127.14.116.193:33655)
I20260619 06:52:18.113751 1016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.193:60063
W20260619 06:52:18.217657 1207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:18.217914 1207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:18.217952 1207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:18.221946 1207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:18.221997 1207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:18.222050 1207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:52:18.226696 1207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:18.227691 1207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:18.229019 1207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:18.238124 1212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:18.238122 1207 server_base.cc:1034] running on GCE node
W20260619 06:52:18.238103 1215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:18.238140 1213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:18.238685 1207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:18.239277 1207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:18.240464 1207 hybrid_clock.cc:648] HybridClock initialized: now 1781851938240449 us; error 26 us; skew 500 ppm
I20260619 06:52:18.242327 1207 webserver.cc:460] Webserver started at http://127.14.116.194:32919/ using document root <none> and password file <none>
I20260619 06:52:18.242888 1207 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:18.242981 1207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:18.243199 1207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:18.245374 1207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "dfccd08730e94f6bbddc9292e4610a2b"
format_stamp: "Formatted at 2026-06-19 06:52:18 on dist-test-slave-9x3b"
server_key: "74f1794defea435cd897a9392cff06d8"
server_key_iv: "63aed3eecd96495d6d8d1f7929a771d2"
server_key_version: "encryptionkey@0"
I20260619 06:52:18.245913 1207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "dfccd08730e94f6bbddc9292e4610a2b"
format_stamp: "Formatted at 2026-06-19 06:52:18 on dist-test-slave-9x3b"
server_key: "74f1794defea435cd897a9392cff06d8"
server_key_iv: "63aed3eecd96495d6d8d1f7929a771d2"
server_key_version: "encryptionkey@0"
I20260619 06:52:18.249351 1207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 06:52:18.251592 1221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:18.252753 1207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:18.252883 1207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "dfccd08730e94f6bbddc9292e4610a2b"
format_stamp: "Formatted at 2026-06-19 06:52:18 on dist-test-slave-9x3b"
server_key: "74f1794defea435cd897a9392cff06d8"
server_key_iv: "63aed3eecd96495d6d8d1f7929a771d2"
server_key_version: "encryptionkey@0"
I20260619 06:52:18.252988 1207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:18.271445 1207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:18.272148 1207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:18.272348 1207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:18.272912 1207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:18.273821 1207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:18.273890 1207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:18.273950 1207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:18.273999 1207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:18.282784 1207 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:42959
I20260619 06:52:18.282811 1334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:42959 every 8 connection(s)
I20260619 06:52:18.283645 1207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:52:18.287645 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 1207
I20260619 06:52:18.287827 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:52:18.288151 14803 external_mini_cluster.cc:1428] Setting key 5edb5367c5c06976f2bd831306d52cf2
I20260619 06:52:18.290189 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46683
--builtin_ntp_servers=127.14.116.212:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:52:18.294150 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.285421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:48409 (local address 127.14.116.254:46683)
0619 06:52:18.285555 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.285558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.286585 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 06:52:18.288291 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.288295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.288364 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.288437 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:18.289503 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.290257 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.292071 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.292300 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.292412 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.293639 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:18.293648 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:18.293650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:18.293662 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:18.293681 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:18.293688 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:18.293763 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.293949 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.294011 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:52:18.295054 1337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.285791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46683 (local address 127.14.116.194:48409)
0619 06:52:18.286444 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:18.286483 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:18.288020 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:18.288633 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:18.288640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:18.288722 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:18.289349 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.289369 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.290413 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.290419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.291953 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.291963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.292451 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.292454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.292666 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.293367 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:52:18.293369 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:18.293410 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:52:18.293473 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:18.293793 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:18.293801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:18.293806 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:52:18.293949 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 06:52:18.294230 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
I20260619 06:52:18.296176 1335 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46683
I20260619 06:52:18.296422 1335 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:18.296911 1335 heartbeater.cc:511] Master 127.14.116.254:46683 requested a full tablet report, sending...
I20260619 06:52:18.297961 1016 ts_manager.cc:194] Registered new tserver with Master: dfccd08730e94f6bbddc9292e4610a2b (127.14.116.194:42959)
I20260619 06:52:18.298817 1016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.194:48409
W20260619 06:52:18.399204 1338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:18.399436 1338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:18.399472 1338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:18.403331 1338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:18.403381 1338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:18.403434 1338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:52:18.408157 1338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:18.409075 1338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:18.410316 1338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:18.419394 1344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:18.419456 1343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:18.419407 1346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:18.419541 1338 server_base.cc:1034] running on GCE node
I20260619 06:52:18.420014 1338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:18.420631 1338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:18.421779 1338 hybrid_clock.cc:648] HybridClock initialized: now 1781851938421746 us; error 45 us; skew 500 ppm
I20260619 06:52:18.423454 1338 webserver.cc:460] Webserver started at http://127.14.116.195:35701/ using document root <none> and password file <none>
I20260619 06:52:18.424038 1338 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:18.424099 1338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:18.424347 1338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:18.426252 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "93a22b3fdfd04c8b94c7ea750beb0743"
format_stamp: "Formatted at 2026-06-19 06:52:18 on dist-test-slave-9x3b"
server_key: "66c1c1ffd0b589d7e1d6344963de8ac5"
server_key_iv: "55942be1fc761d2a368aa84d5eb3b786"
server_key_version: "encryptionkey@0"
I20260619 06:52:18.426796 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "93a22b3fdfd04c8b94c7ea750beb0743"
format_stamp: "Formatted at 2026-06-19 06:52:18 on dist-test-slave-9x3b"
server_key: "66c1c1ffd0b589d7e1d6344963de8ac5"
server_key_iv: "55942be1fc761d2a368aa84d5eb3b786"
server_key_version: "encryptionkey@0"
I20260619 06:52:18.430348 1338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 06:52:18.432767 1352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:18.433887 1338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:52:18.434013 1338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "93a22b3fdfd04c8b94c7ea750beb0743"
format_stamp: "Formatted at 2026-06-19 06:52:18 on dist-test-slave-9x3b"
server_key: "66c1c1ffd0b589d7e1d6344963de8ac5"
server_key_iv: "55942be1fc761d2a368aa84d5eb3b786"
server_key_version: "encryptionkey@0"
I20260619 06:52:18.434118 1338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:18.462738 1338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:18.463296 1338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:18.463438 1338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:18.463994 1338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:18.464819 1338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:18.464872 1338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:18.464910 1338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:18.464931 1338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:18.473160 1338 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:41113
I20260619 06:52:18.473202 1465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:41113 every 8 connection(s)
I20260619 06:52:18.473992 1338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:52:18.476385 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 1338
I20260619 06:52:18.476507 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:52:18.476791 14803 external_mini_cluster.cc:1428] Setting key 4cebebd5fa9fa3fdcbfc1e6349f4a0ef
I20260619 06:52:18.483830 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.475748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:41813 (local address 127.14.116.254:46683)
0619 06:52:18.475882 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.475886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.477103 (+ 1217us) server_negotiation.cc:407] Connection header received
0619 06:52:18.478723 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.478726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.478776 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.478876 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:18.479751 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.480583 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.481941 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.482177 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.482284 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.483339 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:18.483342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:18.483345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:18.483366 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:18.483392 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:18.483401 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:18.483475 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.483641 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.483698 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:52:18.484594 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.476136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:46683 (local address 127.14.116.195:41813)
0619 06:52:18.476946 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:18.476996 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:18.478512 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:18.479046 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:18.479053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:18.479132 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:18.479581 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.479591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.480691 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.480694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.481812 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.481821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.482317 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.482321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.482486 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.483096 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:52:18.483098 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:18.483135 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:52:18.483193 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:18.483488 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:18.483499 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:18.483509 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:52:18.483646 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 06:52:18.483915 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
I20260619 06:52:18.485518 1466 heartbeater.cc:347] Connected to a master server at 127.14.116.254:46683
I20260619 06:52:18.485733 1466 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:18.486195 1466 heartbeater.cc:511] Master 127.14.116.254:46683 requested a full tablet report, sending...
I20260619 06:52:18.487085 1016 ts_manager.cc:194] Registered new tserver with Master: 93a22b3fdfd04c8b94c7ea750beb0743 (127.14.116.195:41113)
I20260619 06:52:18.487917 1016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.195:41813
I20260619 06:52:18.490381 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:52:18.495640 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.492317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.14.116.254:46683)
0619 06:52:18.492408 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.492412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.492611 (+ 199us) server_negotiation.cc:407] Connection header received
0619 06:52:18.492702 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.492704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.492735 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.492795 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:18.493015 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.493551 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.494317 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.494512 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.494583 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.495255 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:18.495257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:18.495259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:18.495269 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:18.495285 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:18.495293 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:18.495390 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.495483 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.495509 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 06:52:18.498196 1016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:52:18.501027 1016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:52:18.514583 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.509601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.14.116.195:41113)
0619 06:52:18.509997 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.510008 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.510030 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:52:18.510100 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.510105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.510272 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.510425 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:18.510667 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.511739 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.513084 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.513850 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.513986 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.514157 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.514316 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.514410 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":125,"threads_started":1}
I20260619 06:52:18.514592 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.509228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:42959 (local address 127.0.0.1:54178)
0619 06:52:18.510395 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:18.510409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:18.510480 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:18.510918 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:18.510921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:18.510937 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:18.511010 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.511016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.512174 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.512177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.513254 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.513260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.514225 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.514231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.514337 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.514363 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:52:18.514430 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 06:52:18.514484 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"thread_start_us":90,"threads_started":1}
I20260619 06:52:18.514592 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.508668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:41113 (local address 127.0.0.1:59870)
0619 06:52:18.509699 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:18.509716 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:18.509804 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:18.510418 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:18.510423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:18.510446 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:18.510555 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.510561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.511856 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.511860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.512968 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.512977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.513987 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.513990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.514110 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.514164 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 06:52:18.514287 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 06:52:18.514440 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":85,"thread_start_us":82,"threads_started":1}
I20260619 06:52:18.515039 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.508692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.14.116.193:33655)
0619 06:52:18.508983 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.508995 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.509488 (+ 493us) server_negotiation.cc:407] Connection header received
0619 06:52:18.509570 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.509575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.509785 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.509935 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:18.510326 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.511514 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.513184 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.513891 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.514540 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.514699 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.514749 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.514853 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
I20260619 06:52:18.515060 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.509381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.14.116.194:42959)
0619 06:52:18.510043 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.510055 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.510573 (+ 518us) server_negotiation.cc:407] Connection header received
0619 06:52:18.510652 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.510656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.510795 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.510936 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:18.511137 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.512046 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.513355 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.513870 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.514001 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.514164 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.514793 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.514896 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":131,"threads_started":1}
I20260619 06:52:18.515592 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.508548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:33655 (local address 127.0.0.1:44952)
0619 06:52:18.509135 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:18.509202 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:18.509331 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:18.510088 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:18.510092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:18.510109 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:18.510201 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.510205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.511657 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.511661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.513082 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.513093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.514017 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.514020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.514129 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.514181 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 06:52:18.514273 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:52:18.514410 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":60,"thread_start_us":71,"threads_started":1}
I20260619 06:52:18.517442 1400 tablet_service.cc:1467] Processing CreateTablet for tablet dbd0844287ec49ff93ddf4e7e115d3c9 (DEFAULT_TABLE table=test-table [id=556a41fa305544d9a8dccae0a03adea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:18.517735 1138 tablet_service.cc:1467] Processing CreateTablet for tablet dbd0844287ec49ff93ddf4e7e115d3c9 (DEFAULT_TABLE table=test-table [id=556a41fa305544d9a8dccae0a03adea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:18.518404 1400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd0844287ec49ff93ddf4e7e115d3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:18.518642 1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd0844287ec49ff93ddf4e7e115d3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:18.519603 1269 tablet_service.cc:1467] Processing CreateTablet for tablet dbd0844287ec49ff93ddf4e7e115d3c9 (DEFAULT_TABLE table=test-table [id=556a41fa305544d9a8dccae0a03adea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:18.520534 1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd0844287ec49ff93ddf4e7e115d3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:18.524653 1483 tablet_bootstrap.cc:492] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e: Bootstrap starting.
I20260619 06:52:18.525596 1484 tablet_bootstrap.cc:492] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: Bootstrap starting.
I20260619 06:52:18.526755 1485 tablet_bootstrap.cc:492] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: Bootstrap starting.
I20260619 06:52:18.526798 1483 tablet_bootstrap.cc:654] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:18.527529 1484 tablet_bootstrap.cc:654] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:18.527755 1483 log.cc:824] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:18.528309 1484 log.cc:824] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:18.528644 1485 tablet_bootstrap.cc:654] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:18.529382 1485 log.cc:824] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:18.530090 1483 tablet_bootstrap.cc:492] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e: No bootstrap required, opened a new log
I20260619 06:52:18.530104 1484 tablet_bootstrap.cc:492] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: No bootstrap required, opened a new log
I20260619 06:52:18.530277 1483 ts_tablet_manager.cc:1397] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 06:52:18.530292 1484 ts_tablet_manager.cc:1397] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:52:18.531667 1485 tablet_bootstrap.cc:492] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: No bootstrap required, opened a new log
I20260619 06:52:18.531836 1485 ts_tablet_manager.cc:1397] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:52:18.533341 1484 raft_consensus.cc:348] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } }
I20260619 06:52:18.533556 1484 raft_consensus.cc:374] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:18.533607 1484 raft_consensus.cc:729] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a22b3fdfd04c8b94c7ea750beb0743, State: Initialized, Role: FOLLOWER
I20260619 06:52:18.534042 1484 consensus_queue.cc:260] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } }
I20260619 06:52:18.534670 1466 heartbeater.cc:503] Master 127.14.116.254:46683 was elected leader, sending a full tablet report...
I20260619 06:52:18.534648 1483 raft_consensus.cc:348] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } }
I20260619 06:52:18.534874 1483 raft_consensus.cc:374] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:18.534924 1483 raft_consensus.cc:729] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd6bdbc761f64bbd8cec652da6052e0e, State: Initialized, Role: FOLLOWER
I20260619 06:52:18.534976 1484 ts_tablet_manager.cc:1428] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:52:18.535359 1483 consensus_queue.cc:260] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } }
I20260619 06:52:18.535452 1485 raft_consensus.cc:348] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } }
I20260619 06:52:18.535717 1485 raft_consensus.cc:374] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:18.535773 1485 raft_consensus.cc:729] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfccd08730e94f6bbddc9292e4610a2b, State: Initialized, Role: FOLLOWER
I20260619 06:52:18.536175 1485 consensus_queue.cc:260] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } }
I20260619 06:52:18.536262 1483 ts_tablet_manager.cc:1428] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260619 06:52:18.536588 1204 heartbeater.cc:503] Master 127.14.116.254:46683 was elected leader, sending a full tablet report...
I20260619 06:52:18.536943 1335 heartbeater.cc:503] Master 127.14.116.254:46683 was elected leader, sending a full tablet report...
I20260619 06:52:18.537073 1485 ts_tablet_manager.cc:1428] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:52:18.545547 1490 raft_consensus.cc:482] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:52:18.545723 1490 raft_consensus.cc:504] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } }
I20260619 06:52:18.546727 1490 leader_election.cc:288] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93a22b3fdfd04c8b94c7ea750beb0743 (127.14.116.195:41113), dfccd08730e94f6bbddc9292e4610a2b (127.14.116.194:42959)
I20260619 06:52:18.550673 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.547291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:38817 (local address 127.14.116.194:42959)
0619 06:52:18.547410 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.547416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.547598 (+ 182us) server_negotiation.cc:407] Connection header received
0619 06:52:18.547721 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.547723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.547761 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.547821 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:18.548149 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.548643 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.549750 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.550114 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.550201 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.550258 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.550480 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.550555 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:52:18.550676 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.547197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:42959 (local address 127.14.116.193:38817)
0619 06:52:18.547491 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:18.547502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:18.547639 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:18.547864 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:18.547867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:18.547882 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:18.548037 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.548043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.548754 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.548760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.549623 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.549631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.550231 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.550236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.550342 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.550373 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:52:18.550447 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:52:18.550523 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":24,"thread_start_us":125,"threads_started":1}
I20260619 06:52:18.550699 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.546932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:41113 (local address 127.14.116.193:60215)
0619 06:52:18.547043 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:18.547059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:18.547141 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:18.547906 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:18.547909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:18.547933 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:18.548057 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.548062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.548774 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.548778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.549626 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:18.549635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.550305 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.550308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:18.550403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.550442 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:52:18.550521 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:52:18.550578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":66}
I20260619 06:52:18.550729 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.547496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:60215 (local address 127.14.116.195:41113)
0619 06:52:18.547648 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.547653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.547667 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:52:18.547712 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.547718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.547763 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.547868 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:18.548191 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.548650 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.549771 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.550189 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.550301 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.550348 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.550552 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.550617 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:52:18.551245 1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0844287ec49ff93ddf4e7e115d3c9" candidate_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfccd08730e94f6bbddc9292e4610a2b" is_pre_election: true
I20260619 06:52:18.551247 1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0844287ec49ff93ddf4e7e115d3c9" candidate_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" is_pre_election: true
I20260619 06:52:18.551513 1420 raft_consensus.cc:2454] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd6bdbc761f64bbd8cec652da6052e0e in term 0.
I20260619 06:52:18.551513 1289 raft_consensus.cc:2454] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd6bdbc761f64bbd8cec652da6052e0e in term 0.
I20260619 06:52:18.551961 1092 leader_election.cc:302] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfccd08730e94f6bbddc9292e4610a2b, fd6bdbc761f64bbd8cec652da6052e0e; no voters:
I20260619 06:52:18.552186 1490 raft_consensus.cc:2781] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:52:18.552265 1490 raft_consensus.cc:482] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:52:18.552309 1490 raft_consensus.cc:3037] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:18.553285 1490 raft_consensus.cc:504] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } }
I20260619 06:52:18.553592 1490 leader_election.cc:288] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [CANDIDATE]: Term 1 election: Requested vote from peers 93a22b3fdfd04c8b94c7ea750beb0743 (127.14.116.195:41113), dfccd08730e94f6bbddc9292e4610a2b (127.14.116.194:42959)
I20260619 06:52:18.553912 1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0844287ec49ff93ddf4e7e115d3c9" candidate_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a22b3fdfd04c8b94c7ea750beb0743"
I20260619 06:52:18.553978 1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0844287ec49ff93ddf4e7e115d3c9" candidate_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfccd08730e94f6bbddc9292e4610a2b"
I20260619 06:52:18.554016 1420 raft_consensus.cc:3037] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:18.554116 1289 raft_consensus.cc:3037] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:18.554997 1420 raft_consensus.cc:2454] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd6bdbc761f64bbd8cec652da6052e0e in term 1.
I20260619 06:52:18.555128 1289 raft_consensus.cc:2454] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd6bdbc761f64bbd8cec652da6052e0e in term 1.
I20260619 06:52:18.555266 1092 leader_election.cc:302] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a22b3fdfd04c8b94c7ea750beb0743, fd6bdbc761f64bbd8cec652da6052e0e; no voters:
I20260619 06:52:18.555431 1490 raft_consensus.cc:2781] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:18.555670 1490 raft_consensus.cc:686] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [term 1 LEADER]: Becoming Leader. State: Replica: fd6bdbc761f64bbd8cec652da6052e0e, State: Running, Role: LEADER
I20260619 06:52:18.556027 1490 consensus_queue.cc:237] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } }
I20260619 06:52:18.559077 1014 catalog_manager.cc:5495] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e reported cstate change: term changed from 0 to 1, leader changed from <none> to fd6bdbc761f64bbd8cec652da6052e0e (127.14.116.193). New cstate: current_term: 1 leader_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6bdbc761f64bbd8cec652da6052e0e" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 33655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 } health_report { overall_health: UNKNOWN } } }
I20260619 06:52:18.576910 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.573307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.14.116.193:33655)
0619 06:52:18.573448 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:18.573452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:18.573466 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:52:18.573648 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:18.573650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:18.573702 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:18.573803 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:18.574050 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.574561 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.575406 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:18.575690 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:18.575801 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:18.576095 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:18.576471 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:18.576630 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:18.576723 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 06:52:18.576757 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:52:18.582439 1289 raft_consensus.cc:1261] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [term 1 FOLLOWER]: Refusing update from remote peer fd6bdbc761f64bbd8cec652da6052e0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:18.582439 1420 raft_consensus.cc:1261] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [term 1 FOLLOWER]: Refusing update from remote peer fd6bdbc761f64bbd8cec652da6052e0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:18.583105 1490 consensus_queue.cc:1035] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfccd08730e94f6bbddc9292e4610a2b" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:18.583334 1493 consensus_queue.cc:1035] T dbd0844287ec49ff93ddf4e7e115d3c9 P fd6bdbc761f64bbd8cec652da6052e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a22b3fdfd04c8b94c7ea750beb0743" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 41113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:18.592578 1496 mvcc.cc:205] Tried to move back new op lower bound from 7298465540426711040 to 7298465540328738816. Current Snapshot: MvccSnapshot[applied={T|T < 7298465540426711040 or (T in {7298465540426711040})}]
I20260619 06:52:18.598780 1014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:18.598923 1014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:18.601260 1014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5bad7255f504fc995beef63f5848e20: Sending DeleteTablet for 3 replicas of tablet dbd0844287ec49ff93ddf4e7e115d3c9
I20260619 06:52:18.601971 1138 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd0844287ec49ff93ddf4e7e115d3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:18 UTC) from {username='slave'} at 127.0.0.1:44952
I20260619 06:52:18.602063 1400 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd0844287ec49ff93ddf4e7e115d3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:18 UTC) from {username='slave'} at 127.0.0.1:59870
I20260619 06:52:18.602280 1269 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd0844287ec49ff93ddf4e7e115d3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:18 UTC) from {username='slave'} at 127.0.0.1:54178
I20260619 06:52:18.602524 1507 tablet_replica.cc:330] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: stopping tablet replica
I20260619 06:52:18.602756 1508 tablet_replica.cc:330] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: stopping tablet replica
I20260619 06:52:18.602825 1507 raft_consensus.cc:2229] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:18.603111 1507 raft_consensus.cc:2258] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:18.603130 1508 raft_consensus.cc:2229] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:18.603219 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 1076
I20260619 06:52:18.603499 1508 raft_consensus.cc:2258] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:18.604842 1508 ts_tablet_manager.cc:1905] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:18.605605 1507 ts_tablet_manager.cc:1905] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:18.607748 1507 ts_tablet_manager.cc:1918] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:18.607849 1507 log.cc:1196] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/dbd0844287ec49ff93ddf4e7e115d3c9
I20260619 06:52:18.608125 1507 ts_tablet_manager.cc:1939] T dbd0844287ec49ff93ddf4e7e115d3c9 P 93a22b3fdfd04c8b94c7ea750beb0743: Deleting consensus metadata
I20260619 06:52:18.608330 1508 ts_tablet_manager.cc:1918] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:18.608424 1508 log.cc:1196] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/dbd0844287ec49ff93ddf4e7e115d3c9
I20260619 06:52:18.608728 1508 ts_tablet_manager.cc:1939] T dbd0844287ec49ff93ddf4e7e115d3c9 P dfccd08730e94f6bbddc9292e4610a2b: Deleting consensus metadata
I20260619 06:52:18.609274 1001 catalog_manager.cc:4841] TS 93a22b3fdfd04c8b94c7ea750beb0743 (127.14.116.195:41113): tablet dbd0844287ec49ff93ddf4e7e115d3c9 (table test-table [id=556a41fa305544d9a8dccae0a03adea0]) successfully deleted
I20260619 06:52:18.609810 1001 catalog_manager.cc:4841] TS dfccd08730e94f6bbddc9292e4610a2b (127.14.116.194:42959): tablet dbd0844287ec49ff93ddf4e7e115d3c9 (table test-table [id=556a41fa305544d9a8dccae0a03adea0]) successfully deleted
W20260619 06:52:18.610873 1002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:33655 (error 108)
I20260619 06:52:18.611559 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 1207
I20260619 06:52:18.611910 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:18.611561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:33655 (local address 127.0.0.1:44974)
0619 06:52:18.611745 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:18.611819 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:33655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 06:52:18.613114 1002 catalog_manager.cc:4584] TS fd6bdbc761f64bbd8cec652da6052e0e (127.14.116.193:33655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbd0844287ec49ff93ddf4e7e115d3c9: Network error: Client connection negotiation failed: client connection to 127.14.116.193:33655: connect: Connection refused (error 111)
I20260619 06:52:18.616624 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 1338
I20260619 06:52:18.621701 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 985
2026-06-19T06:52:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (953 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:18 dist-test-slave-9x3b krb5kdc[1512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:18 dist-test-slave-9x3b krb5kdc[1512](info): set up 2 sockets
Jun 19 06:52:18 dist-test-slave-9x3b krb5kdc[1512](info): commencing operation
krb5kdc: starting...
W20260619 06:52:19.655454 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:19 dist-test-slave-9x3b krb5kdc[1512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:19.768682 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38793
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:38793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:19.876349 1525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:19.876619 1525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:19.876714 1525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:19.880643 1525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:19.880729 1525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:19.880769 1525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:19.880806 1525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:19.880844 1525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:19.886119 1525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:38793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:38793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:19.887266 1525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:19.888285 1525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:19.896459 1530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:19.896503 1525 server_base.cc:1034] running on GCE node
W20260619 06:52:19.896512 1533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:19.896543 1531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:19.897197 1525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:19.898228 1525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:19.899420 1525 hybrid_clock.cc:648] HybridClock initialized: now 1781851939899402 us; error 29 us; skew 500 ppm
Jun 19 06:52:19 dist-test-slave-9x3b krb5kdc[1512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851939, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:19.902036 1525 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:19.902876 1525 webserver.cc:460] Webserver started at http://127.14.116.254:40103/ using document root <none> and password file <none>
I20260619 06:52:19.903445 1525 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:19.903537 1525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:19.903788 1525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:19.905786 1525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "96086431aca140909d36f97c7e053b96"
format_stamp: "Formatted at 2026-06-19 06:52:19 on dist-test-slave-9x3b"
server_key: "9ccbbfbbd49189fe80ebfbf16c8ffe21"
server_key_iv: "1bb45be1055e1f4b25027efffbda69f8"
server_key_version: "encryptionkey@0"
I20260619 06:52:19.906344 1525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "96086431aca140909d36f97c7e053b96"
format_stamp: "Formatted at 2026-06-19 06:52:19 on dist-test-slave-9x3b"
server_key: "9ccbbfbbd49189fe80ebfbf16c8ffe21"
server_key_iv: "1bb45be1055e1f4b25027efffbda69f8"
server_key_version: "encryptionkey@0"
I20260619 06:52:19.909925 1525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:19.912189 1540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:19.913316 1525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:19.913450 1525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "96086431aca140909d36f97c7e053b96"
format_stamp: "Formatted at 2026-06-19 06:52:19 on dist-test-slave-9x3b"
server_key: "9ccbbfbbd49189fe80ebfbf16c8ffe21"
server_key_iv: "1bb45be1055e1f4b25027efffbda69f8"
server_key_version: "encryptionkey@0"
I20260619 06:52:19.913560 1525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:19.923550 1525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:19.926178 1525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:19.926321 1525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:19.932816 1525 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:38793
I20260619 06:52:19.932909 1592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:38793 every 8 connection(s)
I20260619 06:52:19.933773 1525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:19.936452 1593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:19.941531 1593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96: Bootstrap starting.
I20260619 06:52:19.943791 1593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:19.943951 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 1525
I20260619 06:52:19.944113 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:19.944383 14803 external_mini_cluster.cc:1428] Setting key b6e19591febba3d4aac1d1db46a5d40b
I20260619 06:52:19.944625 1593 log.cc:824] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:19.946550 1593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96: No bootstrap required, opened a new log
I20260619 06:52:19.950659 1593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96086431aca140909d36f97c7e053b96" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38793 } }
I20260619 06:52:19.950922 1593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:19.950979 1593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96086431aca140909d36f97c7e053b96, State: Initialized, Role: FOLLOWER
I20260619 06:52:19.951457 1593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96086431aca140909d36f97c7e053b96" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38793 } }
I20260619 06:52:19.951635 1593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:19.951706 1593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:19.951793 1593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:19.952867 1593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96086431aca140909d36f97c7e053b96" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38793 } }
I20260619 06:52:19.953198 1593 leader_election.cc:302] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96086431aca140909d36f97c7e053b96; no voters:
I20260619 06:52:19.953487 1593 leader_election.cc:288] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:19.953586 1598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [term 1 FOLLOWER]: Leader election won for term 1
Jun 19 06:52:19 dist-test-slave-9x3b krb5kdc[1512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:19.953794 1598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [term 1 LEADER]: Becoming Leader. State: Replica: 96086431aca140909d36f97c7e053b96, State: Running, Role: LEADER
I20260619 06:52:19.954109 1598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96086431aca140909d36f97c7e053b96" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38793 } }
I20260619 06:52:19.954697 1593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:19.955502 1600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96086431aca140909d36f97c7e053b96. Latest consensus state: current_term: 1 leader_uuid: "96086431aca140909d36f97c7e053b96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96086431aca140909d36f97c7e053b96" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38793 } } }
I20260619 06:52:19.955667 1600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:19.956130 1599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96086431aca140909d36f97c7e053b96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96086431aca140909d36f97c7e053b96" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 38793 } } }
I20260619 06:52:19.956892 1608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:19.956279 1599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:19.959844 1608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:19.960287 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:19.945428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.14.116.254:38793)
0619 06:52:19.945764 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:19.945781 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:19.945822 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:52:19.946463 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:19.946480 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:19.946824 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:19.947088 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:19.947989 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:19.948925 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:19.950231 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:19.950958 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:19.951122 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:19.954483 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:19.954489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:19.954504 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:19.954533 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:19.956820 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:19.957376 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:19.957383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:19.957391 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:19.957514 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:19.957802 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:19.957805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:19.957807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:19.958279 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:19.958481 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:19.959078 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 06:52:19.959354 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":101,"threads_started":1}
I20260619 06:52:19.965420 1608 catalog_manager.cc:1347] Generated new cluster ID: f16eaff7251a4c76b7168d04b1ce3ede
I20260619 06:52:19.965481 1608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:19.990903 1608 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:19.991876 1608 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:19.997665 1608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96: Generated new TSK 0
I20260619 06:52:19.998327 1608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.116.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
Entry for principal kudu/127.14.116.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab.
I20260619 06:52:20.026305 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38793
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:20.138335 1619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:20.138587 1619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:20.138628 1619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:20.142563 1619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:20.142613 1619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:20.142670 1619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:52:20.147665 1619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.193.keytab
--principal=kudu/127.14.116.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:20.148571 1619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:20.149468 1619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:20.158715 1625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:20.158758 1624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:20.158934 1619 server_base.cc:1034] running on GCE node
W20260619 06:52:20.158941 1627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:20.159353 1619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:20.160082 1619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:20.161259 1619 hybrid_clock.cc:648] HybridClock initialized: now 1781851940161245 us; error 26 us; skew 500 ppm
Jun 19 06:52:20 dist-test-slave-9x3b krb5kdc[1512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851940, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:20.163902 1619 init.cc:377] Logged in from keytab as kudu/127.14.116.193@KRBTEST.COM (short username kudu)
I20260619 06:52:20.164654 1619 webserver.cc:460] Webserver started at http://127.14.116.193:42153/ using document root <none> and password file <none>
I20260619 06:52:20.165148 1619 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:20.165203 1619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:20.165355 1619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:20.167196 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "b97e21e2bffde2b2623bb717cfd9af06"
server_key_iv: "9e3ef2aeb8d7c40288f4e693acd59b1f"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.167732 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "b97e21e2bffde2b2623bb717cfd9af06"
server_key_iv: "9e3ef2aeb8d7c40288f4e693acd59b1f"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.171010 1619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:52:20.173178 1634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.174325 1619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:52:20.174448 1619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "b97e21e2bffde2b2623bb717cfd9af06"
server_key_iv: "9e3ef2aeb8d7c40288f4e693acd59b1f"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.174543 1619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:20.184326 1619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:20.187489 1619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:20.187723 1619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:20.188275 1619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:20.189043 1619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:20.189087 1619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.189152 1619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:20.189182 1619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.197657 1619 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:36323
I20260619 06:52:20.197679 1747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:36323 every 8 connection(s)
I20260619 06:52:20.198504 1619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:52:20.202615 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 1619
I20260619 06:52:20.202744 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:52:20.203055 14803 external_mini_cluster.cc:1428] Setting key 93540bc895d7c89848119d3de5f3852c
Jun 19 06:52:20 dist-test-slave-9x3b krb5kdc[1512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851940, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.193@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.194@KRBTEST.COM; defaulting to no policy
I20260619 06:52:20.213442 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.200267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:44217 (local address 127.14.116.254:38793)
0619 06:52:20.200435 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.200440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.201208 (+ 768us) server_negotiation.cc:407] Connection header received
0619 06:52:20.202498 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.202501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.202560 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.202661 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:20.203981 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.204683 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.206423 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.206822 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.206948 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.209717 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:20.209720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:20.209722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:20.209746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:20.211200 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:20.211830 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:20.211833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:20.211834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:20.211908 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:20.212299 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:20.212301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:20.212302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:20.212472 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:20.212547 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.213199 (+ 652us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.213305 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:52:20.214318 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.200640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38793 (local address 127.14.116.193:44217)
0619 06:52:20.201055 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.201097 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:20.202262 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:20.202862 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:20.202869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:20.203249 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:20.203838 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.203853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.204811 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.204815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.206302 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.206312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.206970 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.206972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.207105 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.207741 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:20.207757 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:20.209547 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:20.211324 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:20.211334 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:20.211349 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:20.211736 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:20.212006 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:20.212011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:20.212014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:20.212204 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:20.212578 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:20.212584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:20.212900 (+ 316us) client_negotiation.cc:769] Sending connection context
0619 06:52:20.213150 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 06:52:20.213448 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.194@KRBTEST.COM" created.
I20260619 06:52:20.215610 1748 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38793
I20260619 06:52:20.215886 1748 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:20.216414 1748 heartbeater.cc:511] Master 127.14.116.254:38793 requested a full tablet report, sending...
I20260619 06:52:20.217916 1557 ts_manager.cc:194] Registered new tserver with Master: f31bdc1d85ca4f0c918d3a3e83ac9bc9 (127.14.116.193:36323)
I20260619 06:52:20.219013 1557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.193@KRBTEST.COM'} at 127.14.116.193:44217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
Entry for principal kudu/127.14.116.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab.
I20260619 06:52:20.230243 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38793
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:20.337136 1753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:20.337371 1753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:20.337410 1753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:20.341287 1753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:20.341336 1753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:20.341390 1753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:52:20.346431 1753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.194.keytab
--principal=kudu/127.14.116.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:20.347316 1753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:20.348217 1753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:20.357371 1758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:20.357365 1759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:20.357362 1761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:20.357712 1753 server_base.cc:1034] running on GCE node
I20260619 06:52:20.358067 1753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:20.358669 1753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:20.359853 1753 hybrid_clock.cc:648] HybridClock initialized: now 1781851940359836 us; error 35 us; skew 500 ppm
Jun 19 06:52:20 dist-test-slave-9x3b krb5kdc[1512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851940, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:20.362351 1753 init.cc:377] Logged in from keytab as kudu/127.14.116.194@KRBTEST.COM (short username kudu)
I20260619 06:52:20.363124 1753 webserver.cc:460] Webserver started at http://127.14.116.194:45693/ using document root <none> and password file <none>
I20260619 06:52:20.363700 1753 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:20.363787 1753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:20.364075 1753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:20.365994 1753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "0cd259a2574a434d8c11200e68c86013"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "59fd664737bc396663f0ff1f1469b6c5"
server_key_iv: "f973bddd712bfe3ef127ce57645e190d"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.366529 1753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "0cd259a2574a434d8c11200e68c86013"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "59fd664737bc396663f0ff1f1469b6c5"
server_key_iv: "f973bddd712bfe3ef127ce57645e190d"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.370041 1753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:20.372291 1768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.373438 1753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:20.373574 1753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "0cd259a2574a434d8c11200e68c86013"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "59fd664737bc396663f0ff1f1469b6c5"
server_key_iv: "f973bddd712bfe3ef127ce57645e190d"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.373679 1753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:20.394502 1753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:20.397346 1753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:20.397540 1753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:20.398119 1753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:20.398931 1753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:20.399000 1753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.399068 1753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:20.399116 1753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.407871 1753 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:46017
I20260619 06:52:20.407889 1881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:46017 every 8 connection(s)
I20260619 06:52:20.408717 1753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:52:20.416239 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 1753
I20260619 06:52:20.416399 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:52:20.416697 14803 external_mini_cluster.cc:1428] Setting key 73d74c6d1d96134c49dad5353e439cef
Jun 19 06:52:20 dist-test-slave-9x3b krb5kdc[1512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851940, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.194@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:20.422308 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.410463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:59377 (local address 127.14.116.254:38793)
0619 06:52:20.410623 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.410628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.411406 (+ 778us) server_negotiation.cc:407] Connection header received
0619 06:52:20.412716 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.412719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.412763 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.412840 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:20.413894 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.414441 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.415492 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.415781 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.415856 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.418538 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:20.418541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:20.418544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:20.418569 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:20.420186 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:20.420670 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:20.420676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:20.420679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:20.420780 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:20.421109 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:20.421114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:20.421119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:20.421364 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:20.421467 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.421923 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.422063 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:52:20.423005 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.410838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38793 (local address 127.14.116.194:59377)
0619 06:52:20.411264 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.411303 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:20.412528 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:20.413000 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:20.413007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:20.413333 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:20.413773 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.413785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.414543 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.414545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.415393 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.415401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.415918 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.415921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.416084 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.416724 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:20.416773 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:20.418384 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:20.420311 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:20.420315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:20.420327 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:20.420575 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:20.420892 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:20.420894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:20.420896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:20.421018 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:20.421461 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:20.421465 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:20.421678 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 06:52:20.421839 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 06:52:20.422083 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20260619 06:52:20.424340 1882 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38793
I20260619 06:52:20.424571 1882 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:20.425031 1882 heartbeater.cc:511] Master 127.14.116.254:38793 requested a full tablet report, sending...
I20260619 06:52:20.426021 1557 ts_manager.cc:194] Registered new tserver with Master: 0cd259a2574a434d8c11200e68c86013 (127.14.116.194:46017)
WARNING: no policy specified for kudu/127.14.116.195@KRBTEST.COM; defaulting to no policy
I20260619 06:52:20.426611 1557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.194@KRBTEST.COM'} at 127.14.116.194:59377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
Entry for principal kudu/127.14.116.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab.
I20260619 06:52:20.442271 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38793
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:20.548017 1887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:20.548243 1887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:20.548281 1887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:20.552153 1887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:20.552205 1887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:20.552260 1887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:52:20.557821 1887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.195.keytab
--principal=kudu/127.14.116.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:38793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:20.558681 1887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:20.559566 1887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:20.568470 1895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:20.568439 1892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:20.568439 1893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:20.568770 1887 server_base.cc:1034] running on GCE node
I20260619 06:52:20.569130 1887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:20.569695 1887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:20.570883 1887 hybrid_clock.cc:648] HybridClock initialized: now 1781851940570874 us; error 27 us; skew 500 ppm
Jun 19 06:52:20 dist-test-slave-9x3b krb5kdc[1512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851940, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:20.573401 1887 init.cc:377] Logged in from keytab as kudu/127.14.116.195@KRBTEST.COM (short username kudu)
I20260619 06:52:20.574124 1887 webserver.cc:460] Webserver started at http://127.14.116.195:39761/ using document root <none> and password file <none>
I20260619 06:52:20.574606 1887 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:20.574661 1887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:20.574806 1887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:20.576659 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "96f7617f7664438ca96d33655304af74"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "8c716cec4d1286ffdb31b637200cf778"
server_key_iv: "93e3bdb621231fa975d946b2876b7d69"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.577144 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "96f7617f7664438ca96d33655304af74"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "8c716cec4d1286ffdb31b637200cf778"
server_key_iv: "93e3bdb621231fa975d946b2876b7d69"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.580649 1887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:52:20.583009 1902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.584241 1887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:20.584348 1887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "96f7617f7664438ca96d33655304af74"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "8c716cec4d1286ffdb31b637200cf778"
server_key_iv: "93e3bdb621231fa975d946b2876b7d69"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.584431 1887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:20.601747 1887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:20.604583 1887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:20.604727 1887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:20.605216 1887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:20.606003 1887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:20.606078 1887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.606146 1887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:20.606170 1887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.614698 1887 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:42273
I20260619 06:52:20.614749 2015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:42273 every 8 connection(s)
I20260619 06:52:20.615538 1887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:52:20.617774 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 1887
I20260619 06:52:20.617913 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:52:20.618194 14803 external_mini_cluster.cc:1428] Setting key a65b46c66738acd5f11b9c1d0a26dd52
Jun 19 06:52:20 dist-test-slave-9x3b krb5kdc[1512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851940, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.195@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:20.628504 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.617309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:41611 (local address 127.14.116.254:38793)
0619 06:52:20.617464 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.617468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.618347 (+ 879us) server_negotiation.cc:407] Connection header received
0619 06:52:20.619834 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.619837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.619880 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.619943 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:20.621070 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.621595 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.622646 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.622848 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.622924 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.625268 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:20.625272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:20.625275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:20.625300 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:20.626492 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:20.626989 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:20.626993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:20.626995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:20.627078 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:20.627407 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:20.627410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:20.627412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:20.627586 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:20.627733 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.628210 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.628325 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:52:20.629207 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.617712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:38793 (local address 127.14.116.195:41611)
0619 06:52:20.618185 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.618235 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:20.619597 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:20.620135 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:20.620143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:20.620483 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:20.620944 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.620957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.621695 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.621698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.622543 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.622551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.622991 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.622994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.623139 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.623697 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:20.623712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:20.625114 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:20.626605 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:20.626611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:20.626623 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:20.626871 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:20.627169 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:20.627173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:20.627175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:20.627306 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:20.627740 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:20.627748 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:20.627981 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 06:52:20.628156 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 06:52:20.628450 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":167,"threads_started":1}
I20260619 06:52:20.630208 2016 heartbeater.cc:347] Connected to a master server at 127.14.116.254:38793
I20260619 06:52:20.630422 2016 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:20.630986 2016 heartbeater.cc:511] Master 127.14.116.254:38793 requested a full tablet report, sending...
I20260619 06:52:20.632102 1557 ts_manager.cc:194] Registered new tserver with Master: 96f7617f7664438ca96d33655304af74 (127.14.116.195:42273)
I20260619 06:52:20.632639 1557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.116.195@KRBTEST.COM'} at 127.14.116.195:41611
I20260619 06:52:20.643404 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:52:20.651819 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.645324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.14.116.254:38793)
0619 06:52:20.645444 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.645447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.645572 (+ 125us) server_negotiation.cc:407] Connection header received
0619 06:52:20.645651 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.645653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.645687 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.645750 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:20.646345 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.646829 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.647586 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.647814 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.647895 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.649067 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:20.649070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:20.649072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:20.649093 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:20.650498 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:20.650834 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:20.650837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:20.650838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:20.650915 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:20.651122 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:20.651124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:20.651125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:20.651313 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:20.651409 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.651571 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.651711 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:52:20.654239 1557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:52:20.657526 1557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:52:20.671044 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.665607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:36323 (local address 127.0.0.1:49152)
0619 06:52:20.666587 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.666600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:20.666695 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:20.667264 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:20.667269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:20.667289 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:20.667389 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.667395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.668775 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.668778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.669718 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.669727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.670541 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.670544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.670641 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.670675 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:52:20.670788 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 06:52:20.670909 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":39,"thread_start_us":92,"threads_started":1}
I20260619 06:52:20.671053 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.665990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.14.116.193:36323)
0619 06:52:20.666368 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.666379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.666675 (+ 296us) server_negotiation.cc:407] Connection header received
0619 06:52:20.666793 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.666798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.666969 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.667113 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:20.667523 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.668631 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.669841 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.670420 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.670558 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.670692 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.670816 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.670909 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":103,"threads_started":1}
I20260619 06:52:20.671522 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.665205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:46017 (local address 127.0.0.1:60130)
0619 06:52:20.665942 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.665990 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:20.666102 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:20.666892 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:20.666897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:20.666917 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:20.667050 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.667060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.668544 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.668548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.669739 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.669750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.671021 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.671025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.671181 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.671226 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:52:20.671292 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 06:52:20.671373 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":78,"thread_start_us":72,"threads_started":1}
I20260619 06:52:20.671578 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.665946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.14.116.195:42273)
0619 06:52:20.666323 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.666334 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.666605 (+ 271us) server_negotiation.cc:407] Connection header received
0619 06:52:20.666779 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.666785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.666969 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.667113 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:20.667302 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.668153 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.669884 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.670717 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.670835 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.670956 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.671342 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.671439 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":79,"threads_started":1}
I20260619 06:52:20.671752 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.665302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:42273 (local address 127.0.0.1:53910)
0619 06:52:20.666515 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.666528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:20.666627 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:20.667089 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:20.667095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:20.667117 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:20.667211 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.667218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.668278 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.668284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.669739 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.669750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.671021 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.671025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.671177 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.671215 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:52:20.671289 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:52:20.671373 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":93,"thread_start_us":95,"threads_started":1}
I20260619 06:52:20.671808 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.665316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.14.116.194:46017)
0619 06:52:20.666392 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.666405 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.666428 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:52:20.666531 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.666537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.666748 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.666899 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:20.667181 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.668410 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.669893 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.670717 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.670835 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.670956 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.671459 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.671563 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973,"thread_start_us":120,"threads_started":1}
I20260619 06:52:20.674183 1682 tablet_service.cc:1467] Processing CreateTablet for tablet 04a69138c135464f868f0f22597ffb0d (DEFAULT_TABLE table=test-table [id=b4bbccb268144892a7d048471e620a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:20.674355 1950 tablet_service.cc:1467] Processing CreateTablet for tablet 04a69138c135464f868f0f22597ffb0d (DEFAULT_TABLE table=test-table [id=b4bbccb268144892a7d048471e620a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:20.674695 1816 tablet_service.cc:1467] Processing CreateTablet for tablet 04a69138c135464f868f0f22597ffb0d (DEFAULT_TABLE table=test-table [id=b4bbccb268144892a7d048471e620a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:20.675235 1682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a69138c135464f868f0f22597ffb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:20.675346 1950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a69138c135464f868f0f22597ffb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:20.675554 1816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a69138c135464f868f0f22597ffb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:20.682363 2035 tablet_bootstrap.cc:492] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013: Bootstrap starting.
I20260619 06:52:20.682834 2033 tablet_bootstrap.cc:492] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9: Bootstrap starting.
I20260619 06:52:20.683917 2034 tablet_bootstrap.cc:492] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: Bootstrap starting.
I20260619 06:52:20.684571 2033 tablet_bootstrap.cc:654] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:20.684571 2035 tablet_bootstrap.cc:654] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:20.685598 2033 log.cc:824] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:20.685796 2035 log.cc:824] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:20.685909 2034 tablet_bootstrap.cc:654] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:20.687191 2034 log.cc:824] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:20.687672 2033 tablet_bootstrap.cc:492] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9: No bootstrap required, opened a new log
I20260619 06:52:20.687881 2033 ts_tablet_manager.cc:1397] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:52:20.689023 2035 tablet_bootstrap.cc:492] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013: No bootstrap required, opened a new log
I20260619 06:52:20.689023 2034 tablet_bootstrap.cc:492] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: No bootstrap required, opened a new log
I20260619 06:52:20.689195 2034 ts_tablet_manager.cc:1397] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:52:20.689195 2035 ts_tablet_manager.cc:1397] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 06:52:20.690851 2033 raft_consensus.cc:348] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } }
I20260619 06:52:20.691072 2033 raft_consensus.cc:374] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:20.691196 2033 raft_consensus.cc:729] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f31bdc1d85ca4f0c918d3a3e83ac9bc9, State: Initialized, Role: FOLLOWER
I20260619 06:52:20.691697 2033 consensus_queue.cc:260] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } }
I20260619 06:52:20.692297 1748 heartbeater.cc:503] Master 127.14.116.254:38793 was elected leader, sending a full tablet report...
I20260619 06:52:20.692636 2033 ts_tablet_manager.cc:1428] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:52:20.692885 2034 raft_consensus.cc:348] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } }
I20260619 06:52:20.692885 2035 raft_consensus.cc:348] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } }
I20260619 06:52:20.693113 2034 raft_consensus.cc:374] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:20.693113 2035 raft_consensus.cc:374] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:20.693162 2035 raft_consensus.cc:729] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd259a2574a434d8c11200e68c86013, State: Initialized, Role: FOLLOWER
I20260619 06:52:20.693171 2034 raft_consensus.cc:729] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f7617f7664438ca96d33655304af74, State: Initialized, Role: FOLLOWER
I20260619 06:52:20.693807 2035 consensus_queue.cc:260] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } }
I20260619 06:52:20.694407 1882 heartbeater.cc:503] Master 127.14.116.254:38793 was elected leader, sending a full tablet report...
I20260619 06:52:20.694749 2035 ts_tablet_manager.cc:1428] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:52:20.695391 2034 consensus_queue.cc:260] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } }
I20260619 06:52:20.696064 2016 heartbeater.cc:503] Master 127.14.116.254:38793 was elected leader, sending a full tablet report...
I20260619 06:52:20.696307 2034 ts_tablet_manager.cc:1428] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
W20260619 06:52:20.699730 1749 tablet.cc:2369] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:52:20.716813 2041 raft_consensus.cc:482] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:52:20.717031 2041 raft_consensus.cc:504] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } }
I20260619 06:52:20.718268 2041 leader_election.cc:288] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cd259a2574a434d8c11200e68c86013 (127.14.116.194:46017), f31bdc1d85ca4f0c918d3a3e83ac9bc9 (127.14.116.193:36323)
I20260619 06:52:20.722770 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.718578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:56027 (local address 127.14.116.194:46017)
0619 06:52:20.718720 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.718725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.718743 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:52:20.718801 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.718807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.718864 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.718946 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:20.719252 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.720048 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.721506 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.722117 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.722247 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.722297 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.722582 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.722651 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:52:20.722770 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.718634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:49999 (local address 127.14.116.193:36323)
0619 06:52:20.718766 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.718772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.719093 (+ 321us) server_negotiation.cc:407] Connection header received
0619 06:52:20.719213 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.719218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.719269 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.719362 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:20.719581 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.720256 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.721597 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.722163 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.722261 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.722299 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.722582 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.722651 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:52:20.722802 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.718455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:46017 (local address 127.14.116.195:56027)
0619 06:52:20.718589 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.718606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:20.718697 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:20.718974 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:20.718977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:20.718993 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:20.719138 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.719146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.720149 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.720152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.721380 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.721391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.722254 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.722258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.722433 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.722475 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:52:20.722552 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:52:20.722664 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":88}
I20260619 06:52:20.722815 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.718455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:36323 (local address 127.14.116.195:49999)
0619 06:52:20.719005 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.719017 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:20.719105 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:20.719370 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:20.719372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:20.719390 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:20.719484 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.719488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.720347 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.720350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.721495 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:20.721505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.722257 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.722259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:20.722429 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.722483 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 06:52:20.722552 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 06:52:20.722664 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":287,"thread_start_us":136,"threads_started":1}
I20260619 06:52:20.723495 1836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a69138c135464f868f0f22597ffb0d" candidate_uuid: "96f7617f7664438ca96d33655304af74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd259a2574a434d8c11200e68c86013" is_pre_election: true
I20260619 06:52:20.723551 1702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a69138c135464f868f0f22597ffb0d" candidate_uuid: "96f7617f7664438ca96d33655304af74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" is_pre_election: true
I20260619 06:52:20.723812 1836 raft_consensus.cc:2454] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f7617f7664438ca96d33655304af74 in term 0.
I20260619 06:52:20.723897 1702 raft_consensus.cc:2454] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f7617f7664438ca96d33655304af74 in term 0.
I20260619 06:52:20.724363 1903 leader_election.cc:302] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0cd259a2574a434d8c11200e68c86013, 96f7617f7664438ca96d33655304af74, f31bdc1d85ca4f0c918d3a3e83ac9bc9; no voters:
I20260619 06:52:20.724646 2041 raft_consensus.cc:2781] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:52:20.724738 2041 raft_consensus.cc:482] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:52:20.724792 2041 raft_consensus.cc:3037] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:20.725807 2041 raft_consensus.cc:504] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } }
I20260619 06:52:20.726073 2041 leader_election.cc:288] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [CANDIDATE]: Term 1 election: Requested vote from peers 0cd259a2574a434d8c11200e68c86013 (127.14.116.194:46017), f31bdc1d85ca4f0c918d3a3e83ac9bc9 (127.14.116.193:36323)
I20260619 06:52:20.726406 1702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a69138c135464f868f0f22597ffb0d" candidate_uuid: "96f7617f7664438ca96d33655304af74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9"
I20260619 06:52:20.726431 1836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a69138c135464f868f0f22597ffb0d" candidate_uuid: "96f7617f7664438ca96d33655304af74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cd259a2574a434d8c11200e68c86013"
I20260619 06:52:20.726528 1836 raft_consensus.cc:3037] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:20.726528 1702 raft_consensus.cc:3037] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:20.727762 1702 raft_consensus.cc:2454] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f7617f7664438ca96d33655304af74 in term 1.
I20260619 06:52:20.727762 1836 raft_consensus.cc:2454] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f7617f7664438ca96d33655304af74 in term 1.
I20260619 06:52:20.728052 1904 leader_election.cc:302] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0cd259a2574a434d8c11200e68c86013, 96f7617f7664438ca96d33655304af74, f31bdc1d85ca4f0c918d3a3e83ac9bc9; no voters:
I20260619 06:52:20.728204 2041 raft_consensus.cc:2781] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:20.728386 2041 raft_consensus.cc:686] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 1 LEADER]: Becoming Leader. State: Replica: 96f7617f7664438ca96d33655304af74, State: Running, Role: LEADER
I20260619 06:52:20.728691 2041 consensus_queue.cc:237] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } }
I20260619 06:52:20.731595 1556 catalog_manager.cc:5495] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96f7617f7664438ca96d33655304af74 (127.14.116.195). New cstate: current_term: 1 leader_uuid: "96f7617f7664438ca96d33655304af74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96f7617f7664438ca96d33655304af74" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 42273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 } health_report { overall_health: UNKNOWN } } }
I20260619 06:52:20.747772 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.744372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.14.116.195:42273)
0619 06:52:20.744485 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:20.744488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:20.744501 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:52:20.744633 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:20.744636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:20.744683 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:20.744791 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:20.744994 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.745515 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.746350 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:20.746586 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:20.746697 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:20.746998 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:20.747373 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:20.747504 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:20.747547 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 06:52:20.747600 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:52:20.752980 1702 raft_consensus.cc:1261] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [term 1 FOLLOWER]: Refusing update from remote peer 96f7617f7664438ca96d33655304af74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:20.752982 1836 raft_consensus.cc:1261] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [term 1 FOLLOWER]: Refusing update from remote peer 96f7617f7664438ca96d33655304af74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:20.753681 2043 consensus_queue.cc:1035] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cd259a2574a434d8c11200e68c86013" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:20.753847 2041 consensus_queue.cc:1035] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f31bdc1d85ca4f0c918d3a3e83ac9bc9" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 36323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:20.762120 2047 mvcc.cc:205] Tried to move back new op lower bound from 7298465549317697536 to 7298465549227462656. Current Snapshot: MvccSnapshot[applied={T|T < 7298465549317697536}]
I20260619 06:52:20.768829 1556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:20.768992 1556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:20.771212 1556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96086431aca140909d36f97c7e053b96: Sending DeleteTablet for 3 replicas of tablet 04a69138c135464f868f0f22597ffb0d
I20260619 06:52:20.771915 1816 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a69138c135464f868f0f22597ffb0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:20 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:60130
I20260619 06:52:20.771930 1950 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a69138c135464f868f0f22597ffb0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:20 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:53910
I20260619 06:52:20.772380 2055 tablet_replica.cc:330] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013: stopping tablet replica
I20260619 06:52:20.772385 1682 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a69138c135464f868f0f22597ffb0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:20 UTC) from {username='kudu', principal='kudu/127.14.116.254@KRBTEST.COM'} at 127.0.0.1:49152
I20260619 06:52:20.772667 2055 raft_consensus.cc:2229] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:20.772796 2057 tablet_replica.cc:330] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9: stopping tablet replica
I20260619 06:52:20.773016 2056 tablet_replica.cc:330] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: stopping tablet replica
I20260619 06:52:20.773113 2057 raft_consensus.cc:2229] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:20.773315 2056 raft_consensus.cc:2229] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:52:20.773460 2057 raft_consensus.cc:2258] T 04a69138c135464f868f0f22597ffb0d P f31bdc1d85ca4f0c918d3a3e83ac9bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:20.773507 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 1619
I20260619 06:52:20.774653 2056 raft_consensus.cc:2258] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:20.776044 2056 ts_tablet_manager.cc:1905] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:20.778864 2056 ts_tablet_manager.cc:1918] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:20.778944 2056 log.cc:1196] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/04a69138c135464f868f0f22597ffb0d
I20260619 06:52:20.779248 2056 ts_tablet_manager.cc:1939] T 04a69138c135464f868f0f22597ffb0d P 96f7617f7664438ca96d33655304af74: Deleting consensus metadata
I20260619 06:52:20.780360 1543 catalog_manager.cc:4841] TS 96f7617f7664438ca96d33655304af74 (127.14.116.195:42273): tablet 04a69138c135464f868f0f22597ffb0d (table test-table [id=b4bbccb268144892a7d048471e620a42]) successfully deleted
I20260619 06:52:20.780472 2055 raft_consensus.cc:2258] T 04a69138c135464f868f0f22597ffb0d P 0cd259a2574a434d8c11200e68c86013 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 06:52:20.780826 1542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:36323 (error 108)
I20260619 06:52:20.781401 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 1753
I20260619 06:52:20.781855 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.781548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:36323 (local address 127.0.0.1:49160)
0619 06:52:20.781709 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.781772 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:36323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 06:52:20.782058 1542 catalog_manager.cc:4584] TS f31bdc1d85ca4f0c918d3a3e83ac9bc9 (127.14.116.193:36323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04a69138c135464f868f0f22597ffb0d: Network error: Client connection negotiation failed: client connection to 127.14.116.193:36323: connect: Connection refused (error 111)
I20260619 06:52:20.788092 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 1887
I20260619 06:52:20.788894 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:20.788311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:46017 (local address 127.0.0.1:60132)
0619 06:52:20.788759 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:20.788817 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.194:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368}
I20260619 06:52:20.793462 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 1525
2026-06-19T06:52:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2172 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T06:52:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:20Z Disabled control of system clock
I20260619 06:52:20.823040 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:37569
--webserver_interface=127.14.116.254
--webserver_port=0
--builtin_ntp_servers=127.14.116.212:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:52:20.946547 2062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:20.946782 2062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:20.946816 2062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:20.950743 2062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:20.950798 2062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:20.950812 2062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:20.950829 2062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:20.950843 2062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:20.956784 2062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:37569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:20.957774 2062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:20.959052 2062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:20.967121 2067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:20.967136 2068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:20.967255 2062 server_base.cc:1034] running on GCE node
W20260619 06:52:20.967121 2070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:20.967962 2062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:20.969007 2062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:20.970180 2062 hybrid_clock.cc:648] HybridClock initialized: now 1781851940970158 us; error 42 us; skew 500 ppm
I20260619 06:52:20.972036 2062 webserver.cc:460] Webserver started at http://127.14.116.254:34697/ using document root <none> and password file <none>
I20260619 06:52:20.972589 2062 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:20.972679 2062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:20.972906 2062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:20.974869 2062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "24b77b4a7c95086102a965657f90cce6"
server_key_iv: "0ff6ce89b43b4d49cc6637629597e75b"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.975414 2062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "24b77b4a7c95086102a965657f90cce6"
server_key_iv: "0ff6ce89b43b4d49cc6637629597e75b"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.978899 2062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:52:20.981248 2076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:20.982369 2062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:52:20.982501 2062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9"
format_stamp: "Formatted at 2026-06-19 06:52:20 on dist-test-slave-9x3b"
server_key: "24b77b4a7c95086102a965657f90cce6"
server_key_iv: "0ff6ce89b43b4d49cc6637629597e75b"
server_key_version: "encryptionkey@0"
I20260619 06:52:20.982606 2062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:20.997220 2062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:20.997859 2062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:20.998051 2062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:21.004885 2062 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:37569
I20260619 06:52:21.004896 2128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:37569 every 8 connection(s)
I20260619 06:52:21.005786 2062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:21.008491 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 2062
I20260619 06:52:21.008646 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:21.008894 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:21.008961 14803 external_mini_cluster.cc:1428] Setting key 0e9d516056bf224b28834f4f55bae6cc
I20260619 06:52:21.016183 2129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9: Bootstrap starting.
I20260619 06:52:21.018709 2129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:21.019235 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.010329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.14.116.254:37569)
0619 06:52:21.011011 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.011030 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.011073 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:52:21.011823 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.011847 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.012347 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.012708 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:21.013155 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.014647 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.015531 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.016222 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.016374 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.017214 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:21.017222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:21.017239 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:21.017716 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:21.017794 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:21.017821 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:21.017915 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.018027 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.018281 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
I20260619 06:52:21.019579 2129 log.cc:824] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:21.021497 2129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9: No bootstrap required, opened a new log
I20260619 06:52:21.024757 2129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37569 } }
I20260619 06:52:21.024956 2129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:21.024991 2129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3d7fb6c74e461ca6e6d998ceb4a7c9, State: Initialized, Role: FOLLOWER
I20260619 06:52:21.025334 2129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37569 } }
I20260619 06:52:21.025439 2129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:21.025503 2129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:21.025602 2129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:21.026460 2129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37569 } }
I20260619 06:52:21.026834 2129 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3d7fb6c74e461ca6e6d998ceb4a7c9; no voters:
I20260619 06:52:21.027207 2129 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:21.027316 2134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:21.027572 2134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [term 1 LEADER]: Becoming Leader. State: Replica: 4e3d7fb6c74e461ca6e6d998ceb4a7c9, State: Running, Role: LEADER
I20260619 06:52:21.027920 2134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37569 } }
I20260619 06:52:21.028381 2129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:21.029536 2136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3d7fb6c74e461ca6e6d998ceb4a7c9. Latest consensus state: current_term: 1 leader_uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37569 } } }
I20260619 06:52:21.029668 2136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:21.029934 2135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3d7fb6c74e461ca6e6d998ceb4a7c9" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37569 } } }
I20260619 06:52:21.030036 2135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:21.030043 2143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:21.033092 2143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:21.038290 2143 catalog_manager.cc:1347] Generated new cluster ID: 17a8b4a5555e4e6aab3519903b3885f8
I20260619 06:52:21.038353 2143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:21.048580 2143 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:21.049712 2143 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:21.054577 2143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9: Generated new TSK 0
I20260619 06:52:21.055322 2143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:21.064741 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.193:0
--local_ip_for_outbound_sockets=127.14.116.193
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:37569
--builtin_ntp_servers=127.14.116.212:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:52:21.187331 2153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:21.187637 2153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:21.187711 2153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:21.191797 2153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:21.191865 2153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:21.191953 2153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.193
I20260619 06:52:21.196620 2153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.116.193
--webserver_port=0
--tserver_master_addrs=127.14.116.254:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.193
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:21.197643 2153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:21.198946 2153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:21.208061 2158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:21.208077 2159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:21.208165 2153 server_base.cc:1034] running on GCE node
W20260619 06:52:21.208077 2161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:21.208636 2153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:21.209254 2153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:21.210436 2153 hybrid_clock.cc:648] HybridClock initialized: now 1781851941210415 us; error 33 us; skew 500 ppm
I20260619 06:52:21.212162 2153 webserver.cc:460] Webserver started at http://127.14.116.193:45447/ using document root <none> and password file <none>
I20260619 06:52:21.212695 2153 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:21.212782 2153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:21.213012 2153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:21.214979 2153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/data/instance:
uuid: "b19b2a75b2e640a6bfef3b1e651f79a0"
format_stamp: "Formatted at 2026-06-19 06:52:21 on dist-test-slave-9x3b"
server_key: "2e9b05075188cf8ad1e9dfb301fa620b"
server_key_iv: "5591f5df2dcf643bcbb586816b29ae7e"
server_key_version: "encryptionkey@0"
I20260619 06:52:21.215521 2153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance:
uuid: "b19b2a75b2e640a6bfef3b1e651f79a0"
format_stamp: "Formatted at 2026-06-19 06:52:21 on dist-test-slave-9x3b"
server_key: "2e9b05075188cf8ad1e9dfb301fa620b"
server_key_iv: "5591f5df2dcf643bcbb586816b29ae7e"
server_key_version: "encryptionkey@0"
I20260619 06:52:21.218989 2153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:52:21.221254 2167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:21.222375 2153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 06:52:21.222504 2153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/wal
uuid: "b19b2a75b2e640a6bfef3b1e651f79a0"
format_stamp: "Formatted at 2026-06-19 06:52:21 on dist-test-slave-9x3b"
server_key: "2e9b05075188cf8ad1e9dfb301fa620b"
server_key_iv: "5591f5df2dcf643bcbb586816b29ae7e"
server_key_version: "encryptionkey@0"
I20260619 06:52:21.222613 2153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:21.234418 2153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:21.235023 2153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:21.235200 2153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:21.235740 2153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:21.236589 2153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:21.236658 2153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:21.236722 2153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:21.236773 2153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:21.245221 2153 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.193:44191
I20260619 06:52:21.245260 2280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.193:44191 every 8 connection(s)
I20260619 06:52:21.246019 2153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/data/info.pb
I20260619 06:52:21.251968 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 2153
I20260619 06:52:21.252105 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-0/wal/instance
I20260619 06:52:21.252409 14803 external_mini_cluster.cc:1428] Setting key 04b12f2d7ba2e5a0fbc3f5992bd04821
I20260619 06:52:21.254479 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.194:0
--local_ip_for_outbound_sockets=127.14.116.194
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:37569
--builtin_ntp_servers=127.14.116.212:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:52:21.255960 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.247737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:36343 (local address 127.14.116.254:37569)
0619 06:52:21.247888 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.247892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.248908 (+ 1016us) server_negotiation.cc:407] Connection header received
0619 06:52:21.250367 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.250371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.250432 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.250557 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:21.251472 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.252194 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.253821 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.254134 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.254277 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.255410 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:21.255417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:21.255422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:21.255439 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:21.255466 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:21.255479 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:21.255565 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.255788 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.255856 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:52:21.257177 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.248112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:37569 (local address 127.14.116.193:36343)
0619 06:52:21.248742 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:21.248784 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:21.250149 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:21.250736 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:21.250743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:21.250826 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:21.251312 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.251325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.252328 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.252331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.253685 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.253697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.254300 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.254303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.254457 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.255149 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:52:21.255151 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:21.255192 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:52:21.255254 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:21.255571 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:21.255578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:21.255582 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:52:21.255778 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 06:52:21.256105 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260619 06:52:21.258430 2281 heartbeater.cc:347] Connected to a master server at 127.14.116.254:37569
I20260619 06:52:21.258692 2281 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:21.259246 2281 heartbeater.cc:511] Master 127.14.116.254:37569 requested a full tablet report, sending...
I20260619 06:52:21.261050 2093 ts_manager.cc:194] Registered new tserver with Master: b19b2a75b2e640a6bfef3b1e651f79a0 (127.14.116.193:44191)
I20260619 06:52:21.262956 2093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.193:36343
W20260619 06:52:21.368405 2284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:21.368649 2284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:21.368685 2284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:21.372574 2284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:21.372617 2284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:21.372668 2284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.194
I20260619 06:52:21.377198 2284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.116.194
--webserver_port=0
--tserver_master_addrs=127.14.116.254:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.194
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:21.378096 2284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:21.379338 2284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:21.388501 2289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:21.388501 2290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:21.388684 2284 server_base.cc:1034] running on GCE node
W20260619 06:52:21.388541 2292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:21.389173 2284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:21.389767 2284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:21.390924 2284 hybrid_clock.cc:648] HybridClock initialized: now 1781851941390892 us; error 49 us; skew 500 ppm
I20260619 06:52:21.392730 2284 webserver.cc:460] Webserver started at http://127.14.116.194:45467/ using document root <none> and password file <none>
I20260619 06:52:21.393273 2284 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:21.393332 2284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:21.393540 2284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:21.395490 2284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/data/instance:
uuid: "9a5f18398ca6488f830927bf6f4dbd44"
format_stamp: "Formatted at 2026-06-19 06:52:21 on dist-test-slave-9x3b"
server_key: "569bb5fcfe57b6aa61765fd6a700f101"
server_key_iv: "4fd52f1db14965d4d997ecef1955ef70"
server_key_version: "encryptionkey@0"
I20260619 06:52:21.396111 2284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance:
uuid: "9a5f18398ca6488f830927bf6f4dbd44"
format_stamp: "Formatted at 2026-06-19 06:52:21 on dist-test-slave-9x3b"
server_key: "569bb5fcfe57b6aa61765fd6a700f101"
server_key_iv: "4fd52f1db14965d4d997ecef1955ef70"
server_key_version: "encryptionkey@0"
I20260619 06:52:21.399499 2284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:52:21.401898 2298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:21.403079 2284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:21.403192 2284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/wal
uuid: "9a5f18398ca6488f830927bf6f4dbd44"
format_stamp: "Formatted at 2026-06-19 06:52:21 on dist-test-slave-9x3b"
server_key: "569bb5fcfe57b6aa61765fd6a700f101"
server_key_iv: "4fd52f1db14965d4d997ecef1955ef70"
server_key_version: "encryptionkey@0"
I20260619 06:52:21.403292 2284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:21.412513 2284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:21.413077 2284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:21.413252 2284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:21.413770 2284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:21.414677 2284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:21.414722 2284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:21.414786 2284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:21.414822 2284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:21.423451 2284 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.194:37549
I20260619 06:52:21.423476 2411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.194:37549 every 8 connection(s)
I20260619 06:52:21.424315 2284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/data/info.pb
I20260619 06:52:21.431882 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 2284
I20260619 06:52:21.432029 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/wal/instance
I20260619 06:52:21.432307 14803 external_mini_cluster.cc:1428] Setting key 7cb19fd6d47d9c804b5c75fc8d2adb2b
I20260619 06:52:21.433713 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.425983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.194:50133 (local address 127.14.116.254:37569)
0619 06:52:21.426119 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.426123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.427180 (+ 1057us) server_negotiation.cc:407] Connection header received
0619 06:52:21.428618 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.428622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.428680 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.428780 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:21.429747 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.430269 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.431393 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.431884 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.432015 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.433218 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:21.433222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:21.433225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:21.433237 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:21.433259 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:21.433269 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:21.433346 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.433488 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.433547 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:52:21.434394 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.116.195:0
--local_ip_for_outbound_sockets=127.14.116.195
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:37569
--builtin_ntp_servers=127.14.116.212:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:52:21.434656 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.426418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:37569 (local address 127.14.116.194:50133)
0619 06:52:21.427034 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:21.427072 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:21.428409 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:21.428942 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:21.428949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:21.429029 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:21.429597 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.429612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.430392 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.430395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.431262 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.431270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.432053 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.432057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.432244 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.432933 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:52:21.432935 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:21.432985 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:52:21.433071 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:21.433360 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:21.433366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:21.433370 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:52:21.433461 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:52:21.433742 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
I20260619 06:52:21.435774 2412 heartbeater.cc:347] Connected to a master server at 127.14.116.254:37569
I20260619 06:52:21.436029 2412 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:21.436681 2412 heartbeater.cc:511] Master 127.14.116.254:37569 requested a full tablet report, sending...
I20260619 06:52:21.437706 2093 ts_manager.cc:194] Registered new tserver with Master: 9a5f18398ca6488f830927bf6f4dbd44 (127.14.116.194:37549)
I20260619 06:52:21.438262 2093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.194:50133
W20260619 06:52:21.542140 2415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:21.542390 2415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:21.542462 2415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:21.546463 2415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:21.546517 2415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:21.546619 2415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.116.195
I20260619 06:52:21.551311 2415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.116.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.116.195
--webserver_port=0
--tserver_master_addrs=127.14.116.254:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.116.195
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:21.552388 2415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:21.553714 2415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:21.562858 2423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:21.562883 2420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:21.563022 2415 server_base.cc:1034] running on GCE node
W20260619 06:52:21.562883 2421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:21.563737 2415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:21.564262 2415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:21.565423 2415 hybrid_clock.cc:648] HybridClock initialized: now 1781851941565410 us; error 22 us; skew 500 ppm
I20260619 06:52:21.569845 2415 webserver.cc:460] Webserver started at http://127.14.116.195:43561/ using document root <none> and password file <none>
I20260619 06:52:21.570406 2415 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:21.570468 2415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:21.570626 2415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:21.572564 2415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/data/instance:
uuid: "bf1149f42ecc415caf0abf9e89315f93"
format_stamp: "Formatted at 2026-06-19 06:52:21 on dist-test-slave-9x3b"
server_key: "b6c061f73d1897cebbdc7711d2b5f62f"
server_key_iv: "6725b8ef1eb85032cea238f8cedf30a6"
server_key_version: "encryptionkey@0"
I20260619 06:52:21.573056 2415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance:
uuid: "bf1149f42ecc415caf0abf9e89315f93"
format_stamp: "Formatted at 2026-06-19 06:52:21 on dist-test-slave-9x3b"
server_key: "b6c061f73d1897cebbdc7711d2b5f62f"
server_key_iv: "6725b8ef1eb85032cea238f8cedf30a6"
server_key_version: "encryptionkey@0"
I20260619 06:52:21.576449 2415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:52:21.578596 2429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:21.579782 2415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:21.579869 2415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/wal
uuid: "bf1149f42ecc415caf0abf9e89315f93"
format_stamp: "Formatted at 2026-06-19 06:52:21 on dist-test-slave-9x3b"
server_key: "b6c061f73d1897cebbdc7711d2b5f62f"
server_key_iv: "6725b8ef1eb85032cea238f8cedf30a6"
server_key_version: "encryptionkey@0"
I20260619 06:52:21.579949 2415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:21.607875 2415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:21.608521 2415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:21.608670 2415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:21.609182 2415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:52:21.610028 2415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:52:21.610074 2415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:21.610112 2415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:52:21.610127 2415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:21.619143 2415 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.195:38065
I20260619 06:52:21.619167 2542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.195:38065 every 8 connection(s)
I20260619 06:52:21.620069 2415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/data/info.pb
I20260619 06:52:21.620826 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 2415
I20260619 06:52:21.620965 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/wal/instance
I20260619 06:52:21.621224 14803 external_mini_cluster.cc:1428] Setting key 9cea4bdd1732bde491f65d3bf89fdc05
I20260619 06:52:21.629220 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.621871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.195:35305 (local address 127.14.116.254:37569)
0619 06:52:21.622000 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.622004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.623252 (+ 1248us) server_negotiation.cc:407] Connection header received
0619 06:52:21.624692 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.624694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.624737 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.624806 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:21.625635 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.626176 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.627337 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.627551 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.627690 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.628763 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:21.628767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:21.628770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:21.628786 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:21.628813 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:21.628830 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:21.628886 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.629061 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.629110 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:52:21.630041 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.622396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:37569 (local address 127.14.116.195:35305)
0619 06:52:21.623101 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:21.623151 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:21.624511 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:21.624937 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:21.624943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:21.625025 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:21.625513 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.625526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.626275 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.626278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.627237 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.627246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.627716 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.627718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.627859 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.628502 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:52:21.628504 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:21.628543 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:52:21.628603 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:21.628898 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:21.628904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:21.628908 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:52:21.629006 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 06:52:21.629218 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":128,"threads_started":1}
I20260619 06:52:21.631119 2543 heartbeater.cc:347] Connected to a master server at 127.14.116.254:37569
I20260619 06:52:21.631357 2543 heartbeater.cc:464] Registering TS with master...
I20260619 06:52:21.631897 2543 heartbeater.cc:511] Master 127.14.116.254:37569 requested a full tablet report, sending...
I20260619 06:52:21.632824 2093 ts_manager.cc:194] Registered new tserver with Master: bf1149f42ecc415caf0abf9e89315f93 (127.14.116.195:38065)
I20260619 06:52:21.633332 2093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.116.195:35305
I20260619 06:52:21.635079 14803 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:52:21.640240 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.637017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.14.116.254:37569)
0619 06:52:21.637125 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.637129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.637258 (+ 129us) server_negotiation.cc:407] Connection header received
0619 06:52:21.637349 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.637351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.637385 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.637450 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:21.637683 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.638167 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.638926 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.639119 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.639189 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.639927 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:21.639930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:21.639932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:21.639942 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:21.639958 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:21.639965 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:21.640021 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.640107 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.640149 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:52:21.642694 2093 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:52:21.645644 2093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:52:21.659351 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.653912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.14.116.195:38065)
0619 06:52:21.654553 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.654565 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.654587 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:52:21.654658 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.654666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.654857 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.655026 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:21.655394 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.656529 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.658042 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.658581 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.658720 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.658899 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.659114 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.659205 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":88,"threads_started":1}
I20260619 06:52:21.659346 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.653422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:38065 (local address 127.0.0.1:54756)
0619 06:52:21.654056 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:21.654104 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:21.654257 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:21.655030 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:21.655036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:21.655068 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:21.655274 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.655280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.656646 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.656649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.657912 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.657922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.658721 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.658725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.658901 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.658958 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 06:52:21.659071 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 06:52:21.659191 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":198,"spinlock_wait_cycles":11520,"thread_start_us":95,"threads_started":1}
I20260619 06:52:21.659974 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.653937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:44191 (local address 127.0.0.1:60506)
0619 06:52:21.654688 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:21.654701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:21.654788 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:21.655572 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:21.655575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:21.655592 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:21.655696 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.655702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.657001 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.657004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.657955 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.657965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.658703 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.658706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.659682 (+ 976us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.659710 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:52:21.659781 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 06:52:21.659853 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":33,"thread_start_us":79,"threads_started":1}
I20260619 06:52:21.660203 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.653294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:37549 (local address 127.0.0.1:37520)
0619 06:52:21.654055 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:21.654101 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:21.654257 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:21.655031 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:21.655038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:21.655071 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:21.655274 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.655281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.656412 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.656418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.657912 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.657922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.659794 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.659798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.659879 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.659895 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 06:52:21.659986 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:52:21.660065 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":124,"spinlock_wait_cycles":25600,"thread_start_us":60,"threads_started":1}
I20260619 06:52:21.660370 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.654346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.14.116.193:44191)
0619 06:52:21.655020 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.655031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.655051 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:52:21.655122 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.655127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.655292 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.655438 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:21.655815 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.656876 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.658083 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.658580 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.658707 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.658897 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.660126 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.660220 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":109,"threads_started":1}
I20260619 06:52:21.660769 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.653443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.14.116.194:37549)
0619 06:52:21.653737 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.653751 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.654408 (+ 657us) server_negotiation.cc:407] Connection header received
0619 06:52:21.654495 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.654500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.654705 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.654868 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:21.655413 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.656304 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.658910 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.659437 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.659572 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.659735 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.660168 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.660415 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":111,"threads_started":1}
I20260619 06:52:21.662477 2477 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc7e7f1f2314e79b04b5623ed9efa88 (DEFAULT_TABLE table=test-table [id=064548051deb4abf84d2c5f493f46f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:21.662843 2215 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc7e7f1f2314e79b04b5623ed9efa88 (DEFAULT_TABLE table=test-table [id=064548051deb4abf84d2c5f493f46f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:21.663472 2477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc7e7f1f2314e79b04b5623ed9efa88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:21.663801 2215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc7e7f1f2314e79b04b5623ed9efa88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:21.669174 2560 tablet_bootstrap.cc:492] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0: Bootstrap starting.
I20260619 06:52:21.669195 2561 tablet_bootstrap.cc:492] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: Bootstrap starting.
I20260619 06:52:21.671396 2560 tablet_bootstrap.cc:654] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:21.671746 2561 tablet_bootstrap.cc:654] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:21.671525 2346 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc7e7f1f2314e79b04b5623ed9efa88 (DEFAULT_TABLE table=test-table [id=064548051deb4abf84d2c5f493f46f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:52:21.672255 2560 log.cc:824] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:21.672504 2561 log.cc:824] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:21.672536 2346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc7e7f1f2314e79b04b5623ed9efa88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:21.674057 2560 tablet_bootstrap.cc:492] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0: No bootstrap required, opened a new log
I20260619 06:52:21.674266 2560 ts_tablet_manager.cc:1397] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260619 06:52:21.675392 2561 tablet_bootstrap.cc:492] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: No bootstrap required, opened a new log
I20260619 06:52:21.675573 2561 ts_tablet_manager.cc:1397] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260619 06:52:21.677683 2560 raft_consensus.cc:348] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } }
I20260619 06:52:21.677932 2560 raft_consensus.cc:374] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:21.677986 2560 raft_consensus.cc:729] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19b2a75b2e640a6bfef3b1e651f79a0, State: Initialized, Role: FOLLOWER
I20260619 06:52:21.678704 2560 consensus_queue.cc:260] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } }
I20260619 06:52:21.679375 2281 heartbeater.cc:503] Master 127.14.116.254:37569 was elected leader, sending a full tablet report...
I20260619 06:52:21.679762 2560 ts_tablet_manager.cc:1428] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 06:52:21.679705 2561 raft_consensus.cc:348] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } }
I20260619 06:52:21.679955 2561 raft_consensus.cc:374] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:21.680018 2561 raft_consensus.cc:729] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1149f42ecc415caf0abf9e89315f93, State: Initialized, Role: FOLLOWER
I20260619 06:52:21.680100 2564 tablet_bootstrap.cc:492] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: Bootstrap starting.
I20260619 06:52:21.680460 2561 consensus_queue.cc:260] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } }
I20260619 06:52:21.681334 2543 heartbeater.cc:503] Master 127.14.116.254:37569 was elected leader, sending a full tablet report...
I20260619 06:52:21.681630 2561 ts_tablet_manager.cc:1428] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260619 06:52:21.682096 2565 raft_consensus.cc:482] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:52:21.682287 2564 tablet_bootstrap.cc:654] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:21.682231 2565 raft_consensus.cc:504] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } }
I20260619 06:52:21.683153 2564 log.cc:824] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:21.683342 2565 leader_election.cc:288] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a5f18398ca6488f830927bf6f4dbd44 (127.14.116.194:37549), bf1149f42ecc415caf0abf9e89315f93 (127.14.116.195:38065)
I20260619 06:52:21.686373 2564 tablet_bootstrap.cc:492] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: No bootstrap required, opened a new log
I20260619 06:52:21.686570 2564 ts_tablet_manager.cc:1397] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 06:52:21.688555 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.684291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:49389 (local address 127.14.116.195:38065)
0619 06:52:21.684449 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.684453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.684609 (+ 156us) server_negotiation.cc:407] Connection header received
0619 06:52:21.684790 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.684794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.684852 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.684905 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:21.685281 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.685969 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.687471 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.687993 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.688117 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.688165 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.688370 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.688447 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:52:21.688578 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.683928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.195:38065 (local address 127.14.116.193:49389)
0619 06:52:21.684528 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:21.684541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:21.684672 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:21.685033 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:21.685036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:21.685051 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:21.685165 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.685171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.686105 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.686108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.687352 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.687362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.688118 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.688121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.688219 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.688247 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:52:21.688354 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:52:21.688430 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":28,"thread_start_us":94,"threads_started":1}
I20260619 06:52:21.688905 2283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.683839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.194:37549 (local address 127.14.116.193:34989)
0619 06:52:21.684073 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:21.684092 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:21.684181 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:21.684760 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:21.684763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:21.684779 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:52:21.684888 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.684896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.686335 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.686340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.687567 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:21.687577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.688508 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.688511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:21.688617 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.688638 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:52:21.688706 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:52:21.688769 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":35}
I20260619 06:52:21.689265 2497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc7e7f1f2314e79b04b5623ed9efa88" candidate_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1149f42ecc415caf0abf9e89315f93" is_pre_election: true
I20260619 06:52:21.689306 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.684313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.116.193:34989 (local address 127.14.116.194:37549)
0619 06:52:21.684537 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.684543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.684555 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:52:21.684611 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.684614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.684659 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.684736 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:52:21.684983 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.686231 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.687692 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.688288 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.688402 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.688447 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.689125 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.689202 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260619 06:52:21.689535 2497 raft_consensus.cc:2454] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19b2a75b2e640a6bfef3b1e651f79a0 in term 0.
I20260619 06:52:21.689808 2366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc7e7f1f2314e79b04b5623ed9efa88" candidate_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a5f18398ca6488f830927bf6f4dbd44" is_pre_election: true
I20260619 06:52:21.690025 2170 leader_election.cc:302] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b19b2a75b2e640a6bfef3b1e651f79a0, bf1149f42ecc415caf0abf9e89315f93; no voters:
I20260619 06:52:21.690302 2565 raft_consensus.cc:2781] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:52:21.690385 2565 raft_consensus.cc:482] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:52:21.690441 2565 raft_consensus.cc:3037] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 0 FOLLOWER]: Advancing to term 1
W20260619 06:52:21.690608 2169 leader_election.cc:341] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a5f18398ca6488f830927bf6f4dbd44 (127.14.116.194:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260619 06:52:21.690765 2564 raft_consensus.cc:348] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } }
I20260619 06:52:21.690987 2564 raft_consensus.cc:374] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:21.691068 2564 raft_consensus.cc:729] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5f18398ca6488f830927bf6f4dbd44, State: Initialized, Role: FOLLOWER
I20260619 06:52:21.691429 2564 consensus_queue.cc:260] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } }
I20260619 06:52:21.691510 2565 raft_consensus.cc:504] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } }
I20260619 06:52:21.691895 2565 leader_election.cc:288] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [CANDIDATE]: Term 1 election: Requested vote from peers 9a5f18398ca6488f830927bf6f4dbd44 (127.14.116.194:37549), bf1149f42ecc415caf0abf9e89315f93 (127.14.116.195:38065)
I20260619 06:52:21.692090 2412 heartbeater.cc:503] Master 127.14.116.254:37569 was elected leader, sending a full tablet report...
I20260619 06:52:21.692185 2366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc7e7f1f2314e79b04b5623ed9efa88" candidate_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a5f18398ca6488f830927bf6f4dbd44"
I20260619 06:52:21.692343 2366 raft_consensus.cc:3037] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:21.692422 2497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc7e7f1f2314e79b04b5623ed9efa88" candidate_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1149f42ecc415caf0abf9e89315f93"
I20260619 06:52:21.692451 2564 ts_tablet_manager.cc:1428] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:52:21.692521 2497 raft_consensus.cc:3037] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:21.693576 2366 raft_consensus.cc:2454] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19b2a75b2e640a6bfef3b1e651f79a0 in term 1.
I20260619 06:52:21.693679 2497 raft_consensus.cc:2454] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19b2a75b2e640a6bfef3b1e651f79a0 in term 1.
I20260619 06:52:21.694070 2170 leader_election.cc:302] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b19b2a75b2e640a6bfef3b1e651f79a0, bf1149f42ecc415caf0abf9e89315f93; no voters:
I20260619 06:52:21.694219 2565 raft_consensus.cc:2781] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:21.694479 2565 raft_consensus.cc:686] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [term 1 LEADER]: Becoming Leader. State: Replica: b19b2a75b2e640a6bfef3b1e651f79a0, State: Running, Role: LEADER
I20260619 06:52:21.694774 2565 consensus_queue.cc:237] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } }
I20260619 06:52:21.697511 2093 catalog_manager.cc:5495] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b19b2a75b2e640a6bfef3b1e651f79a0 (127.14.116.193). New cstate: current_term: 1 leader_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19b2a75b2e640a6bfef3b1e651f79a0" member_type: VOTER last_known_addr { host: "127.14.116.193" port: 44191 } health_report { overall_health: HEALTHY } } }
I20260619 06:52:21.711585 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.708579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.14.116.193:44191)
0619 06:52:21.708693 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:21.708696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:21.708707 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:52:21.708835 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:21.708837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:21.708885 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:21.708966 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:21.709158 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.709638 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.710414 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:21.710610 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:21.710680 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:21.710917 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:21.711285 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:21.711378 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:21.711442 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 06:52:21.711486 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:52:21.717058 2366 raft_consensus.cc:1261] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44 [term 1 FOLLOWER]: Refusing update from remote peer b19b2a75b2e640a6bfef3b1e651f79a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:21.717100 2497 raft_consensus.cc:1261] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [term 1 FOLLOWER]: Refusing update from remote peer b19b2a75b2e640a6bfef3b1e651f79a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:21.717696 2570 consensus_queue.cc:1035] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5f18398ca6488f830927bf6f4dbd44" member_type: VOTER last_known_addr { host: "127.14.116.194" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:21.717914 2565 consensus_queue.cc:1035] T 2bc7e7f1f2314e79b04b5623ed9efa88 P b19b2a75b2e640a6bfef3b1e651f79a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1149f42ecc415caf0abf9e89315f93" member_type: VOTER last_known_addr { host: "127.14.116.195" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:21.724774 2573 mvcc.cc:205] Tried to move back new op lower bound from 7298465553266290688 to 7298465553184354304. Current Snapshot: MvccSnapshot[applied={T|T < 7298465553266290688}]
I20260619 06:52:21.725368 2574 mvcc.cc:205] Tried to move back new op lower bound from 7298465553266290688 to 7298465553184354304. Current Snapshot: MvccSnapshot[applied={T|T < 7298465553266290688}]
I20260619 06:52:21.735261 2093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:21.735459 2093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:52:21.737859 2093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e3d7fb6c74e461ca6e6d998ceb4a7c9: Sending DeleteTablet for 3 replicas of tablet 2bc7e7f1f2314e79b04b5623ed9efa88
I20260619 06:52:21.738556 2346 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc7e7f1f2314e79b04b5623ed9efa88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:21 UTC) from {username='slave'} at 127.0.0.1:37520
I20260619 06:52:21.738698 2477 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc7e7f1f2314e79b04b5623ed9efa88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:21 UTC) from {username='slave'} at 127.0.0.1:54756
I20260619 06:52:21.739022 2583 tablet_replica.cc:330] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: stopping tablet replica
I20260619 06:52:21.739195 2215 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc7e7f1f2314e79b04b5623ed9efa88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:52:21 UTC) from {username='slave'} at 127.0.0.1:60506
I20260619 06:52:21.739450 2583 raft_consensus.cc:2229] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:21.739691 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 2153
I20260619 06:52:21.740785 2583 raft_consensus.cc:2258] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:21.741315 2584 tablet_replica.cc:330] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: stopping tablet replica
I20260619 06:52:21.741613 2584 raft_consensus.cc:2229] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:52:21.741891 2584 raft_consensus.cc:2258] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:52:21.742328 2583 ts_tablet_manager.cc:1905] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:21.743001 2584 ts_tablet_manager.cc:1905] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:52:21.744930 2583 ts_tablet_manager.cc:1918] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:21.745021 2583 log.cc:1196] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-1/wal/wals/2bc7e7f1f2314e79b04b5623ed9efa88
I20260619 06:52:21.745085 2584 ts_tablet_manager.cc:1918] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:52:21.745213 2584 log.cc:1196] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: Deleting WAL directory at /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781851695409368-14803-0/minicluster-data/ts-2/wal/wals/2bc7e7f1f2314e79b04b5623ed9efa88
I20260619 06:52:21.745301 2583 ts_tablet_manager.cc:1939] T 2bc7e7f1f2314e79b04b5623ed9efa88 P 9a5f18398ca6488f830927bf6f4dbd44: Deleting consensus metadata
I20260619 06:52:21.745535 2584 ts_tablet_manager.cc:1939] T 2bc7e7f1f2314e79b04b5623ed9efa88 P bf1149f42ecc415caf0abf9e89315f93: Deleting consensus metadata
I20260619 06:52:21.746421 2078 catalog_manager.cc:4841] TS 9a5f18398ca6488f830927bf6f4dbd44 (127.14.116.194:37549): tablet 2bc7e7f1f2314e79b04b5623ed9efa88 (table test-table [id=064548051deb4abf84d2c5f493f46f3c]) successfully deleted
W20260619 06:52:21.746680 2079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.116.193:44191 (error 108)
I20260619 06:52:21.746847 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 2284
I20260619 06:52:21.747131 2079 catalog_manager.cc:4841] TS bf1149f42ecc415caf0abf9e89315f93 (127.14.116.195:38065): tablet 2bc7e7f1f2314e79b04b5623ed9efa88 (table test-table [id=064548051deb4abf84d2c5f493f46f3c]) successfully deleted
I20260619 06:52:21.748088 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:21.747760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.193:44191 (local address 127.0.0.1:60528)
0619 06:52:21.747912 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:21.747992 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.193:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 06:52:21.748237 2079 catalog_manager.cc:4584] TS b19b2a75b2e640a6bfef3b1e651f79a0 (127.14.116.193:44191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2bc7e7f1f2314e79b04b5623ed9efa88: Network error: Client connection negotiation failed: client connection to 127.14.116.193:44191: connect: Connection refused (error 111)
I20260619 06:52:21.752478 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 2415
I20260619 06:52:21.757583 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 2062
2026-06-19T06:52:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (962 ms)
[----------] 6 tests from EncryptionPolicyTest (9670 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:21 dist-test-slave-9x3b krb5kdc[2589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:21 dist-test-slave-9x3b krb5kdc[2589](info): set up 2 sockets
Jun 19 06:52:21 dist-test-slave-9x3b krb5kdc[2589](info): commencing operation
krb5kdc: starting...
W20260619 06:52:22.783749 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:22 dist-test-slave-9x3b krb5kdc[2589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:22.896523 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:23.004343 2602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:23.004592 2602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:23.004631 2602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:23.008524 2602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:23.008576 2602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:23.008596 2602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:23.008612 2602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:23.008627 2602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:23.013753 2602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45743
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:23.014699 2602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:23.015671 2602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:23.023742 2610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:23.023759 2608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:23.023849 2607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:23.024787 2602 server_base.cc:1034] running on GCE node
I20260619 06:52:23.025342 2602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:23.026371 2602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:23.027578 2602 hybrid_clock.cc:648] HybridClock initialized: now 1781851943027564 us; error 33 us; skew 500 ppm
Jun 19 06:52:23 dist-test-slave-9x3b krb5kdc[2589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:23.030269 2602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:23.031054 2602 webserver.cc:460] Webserver started at http://127.0.0.1:43899/ using document root <none> and password file <none>
I20260619 06:52:23.031533 2602 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:23.031589 2602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:23.031821 2602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:23.033691 2602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "df7a556cdb614d3abec49e7b6c37552e"
format_stamp: "Formatted at 2026-06-19 06:52:23 on dist-test-slave-9x3b"
server_key: "4cf7786a7af393f625e50b1361e33b76"
server_key_iv: "ec05acf1f9070558ac53629d64df1664"
server_key_version: "encryptionkey@0"
I20260619 06:52:23.034188 2602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "df7a556cdb614d3abec49e7b6c37552e"
format_stamp: "Formatted at 2026-06-19 06:52:23 on dist-test-slave-9x3b"
server_key: "4cf7786a7af393f625e50b1361e33b76"
server_key_iv: "ec05acf1f9070558ac53629d64df1664"
server_key_version: "encryptionkey@0"
I20260619 06:52:23.037616 2602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:23.039774 2617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:23.040948 2602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:23.041044 2602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "df7a556cdb614d3abec49e7b6c37552e"
format_stamp: "Formatted at 2026-06-19 06:52:23 on dist-test-slave-9x3b"
server_key: "4cf7786a7af393f625e50b1361e33b76"
server_key_iv: "ec05acf1f9070558ac53629d64df1664"
server_key_version: "encryptionkey@0"
I20260619 06:52:23.041124 2602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:23.060714 2602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:23.064857 2602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:23.065062 2602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:23.073189 2602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45743
I20260619 06:52:23.073212 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45743 every 8 connection(s)
I20260619 06:52:23.074079 2602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:23.076807 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:23.082042 2670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e: Bootstrap starting.
I20260619 06:52:23.082161 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 2602
I20260619 06:52:23.082304 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:23.082613 14803 external_mini_cluster.cc:1428] Setting key 66dd524050d9b9dc0fcf21394bc9115c
I20260619 06:52:23.084630 2670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:23.085325 2670 log.cc:824] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:23.087051 2670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e: No bootstrap required, opened a new log
I20260619 06:52:23.090471 2670 raft_consensus.cc:348] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7a556cdb614d3abec49e7b6c37552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45743 } }
I20260619 06:52:23.090687 2670 raft_consensus.cc:374] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:23.090760 2670 raft_consensus.cc:729] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7a556cdb614d3abec49e7b6c37552e, State: Initialized, Role: FOLLOWER
Jun 19 06:52:23 dist-test-slave-9x3b krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:52:23.091233 2670 consensus_queue.cc:260] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7a556cdb614d3abec49e7b6c37552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45743 } }
I20260619 06:52:23.091356 2670 raft_consensus.cc:388] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:23.091435 2670 raft_consensus.cc:482] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:23.091562 2670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:23.092733 2670 raft_consensus.cc:504] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7a556cdb614d3abec49e7b6c37552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45743 } }
I20260619 06:52:23.093060 2670 leader_election.cc:302] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df7a556cdb614d3abec49e7b6c37552e; no voters:
I20260619 06:52:23.093350 2670 leader_election.cc:288] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:23.093461 2675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:23.093657 2675 raft_consensus.cc:686] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [term 1 LEADER]: Becoming Leader. State: Replica: df7a556cdb614d3abec49e7b6c37552e, State: Running, Role: LEADER
I20260619 06:52:23.093983 2675 consensus_queue.cc:237] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7a556cdb614d3abec49e7b6c37552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45743 } }
I20260619 06:52:23.094583 2670 sys_catalog.cc:564] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:23.095149 2677 sys_catalog.cc:455] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [sys.catalog]: SysCatalogTable state changed. Reason: New leader df7a556cdb614d3abec49e7b6c37552e. Latest consensus state: current_term: 1 leader_uuid: "df7a556cdb614d3abec49e7b6c37552e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7a556cdb614d3abec49e7b6c37552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45743 } } }
I20260619 06:52:23.095376 2677 sys_catalog.cc:458] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:23.095633 2676 sys_catalog.cc:455] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df7a556cdb614d3abec49e7b6c37552e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df7a556cdb614d3abec49e7b6c37552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45743 } } }
I20260619 06:52:23.095755 2676 sys_catalog.cc:458] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:23.096134 2684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:23.097470 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:23.083983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53579 (local address 127.0.0.1:45743)
0619 06:52:23.084420 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:23.084434 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:23.084467 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:52:23.085130 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:23.085148 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:23.085524 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:23.085831 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:23.086580 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:23.087569 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:23.088475 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:23.088940 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:23.089107 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:23.091772 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:23.091777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:23.091789 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:23.091813 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:23.093678 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:23.094332 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:23.094337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:23.094343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:23.094484 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:23.094873 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:23.094876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:23.094878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:23.095427 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:23.095677 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:23.096211 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 06:52:23.096467 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":123,"threads_started":1}
I20260619 06:52:23.099483 2684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:23.100909 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:23.107739 2684 catalog_manager.cc:1347] Generated new cluster ID: 4268697a369a4afbbdd53912b1f94991
I20260619 06:52:23.107816 2684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:23.111559 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:23.103102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36495 (local address 127.0.0.1:45743)
0619 06:52:23.103205 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:23.103209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:23.103396 (+ 187us) server_negotiation.cc:407] Connection header received
0619 06:52:23.103461 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:23.103464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:23.103508 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:23.103595 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:23.104445 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:23.105297 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:23.106319 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:23.106561 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:23.106669 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:23.108143 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:23.108147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:23.108150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:23.108175 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:23.109798 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:23.110292 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:23.110295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:23.110297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:23.110382 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:23.110695 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:23.110699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:23.110700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:23.110919 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:23.111087 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:23.111321 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 06:52:23.111414 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:52:23.121876 2684 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:23.122712 2684 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:23.128340 2684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df7a556cdb614d3abec49e7b6c37552e: Generated new TSK 0
I20260619 06:52:23.128983 2684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:23.136057 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 2602
W20260619 06:52:23.141157 2587 connection.cc:535] client connection to 127.0.0.1:45743 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45743): Connection reset by peer (error 104)
2026-06-19T06:52:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:23 dist-test-slave-9x3b krb5kdc[2701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:23 dist-test-slave-9x3b krb5kdc[2701](info): set up 2 sockets
Jun 19 06:52:23 dist-test-slave-9x3b krb5kdc[2701](info): commencing operation
krb5kdc: starting...
W20260619 06:52:24.167723 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:24 dist-test-slave-9x3b krb5kdc[2701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:24.280395 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:24.387063 2714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:24.387365 2714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:24.387439 2714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:24.391423 2714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:24.391520 2714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:24.391558 2714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:24.391594 2714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:24.391642 2714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:24.396886 2714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:24.397888 2714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:24.398890 2714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:24.407050 2722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:24.407137 2714 server_base.cc:1034] running on GCE node
W20260619 06:52:24.407021 2719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:24.407021 2720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:24.407953 2714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:24.409001 2714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:24.410182 2714 hybrid_clock.cc:648] HybridClock initialized: now 1781851944410174 us; error 32 us; skew 500 ppm
Jun 19 06:52:24 dist-test-slave-9x3b krb5kdc[2701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:24.412946 2714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:24.413820 2714 webserver.cc:460] Webserver started at http://127.0.0.1:36983/ using document root <none> and password file <none>
I20260619 06:52:24.414381 2714 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:24.414471 2714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:24.414698 2714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:24.416769 2714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "8ebea0305c4a4e678edf102bb0a1d659"
format_stamp: "Formatted at 2026-06-19 06:52:24 on dist-test-slave-9x3b"
server_key: "0cb74fb510d849c5123d8e85def97ab3"
server_key_iv: "33ef9f970bbdfeb95178f33325c22157"
server_key_version: "encryptionkey@0"
I20260619 06:52:24.417342 2714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "8ebea0305c4a4e678edf102bb0a1d659"
format_stamp: "Formatted at 2026-06-19 06:52:24 on dist-test-slave-9x3b"
server_key: "0cb74fb510d849c5123d8e85def97ab3"
server_key_iv: "33ef9f970bbdfeb95178f33325c22157"
server_key_version: "encryptionkey@0"
I20260619 06:52:24.420892 2714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:52:24.423189 2729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:24.424340 2714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:24.424484 2714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "8ebea0305c4a4e678edf102bb0a1d659"
format_stamp: "Formatted at 2026-06-19 06:52:24 on dist-test-slave-9x3b"
server_key: "0cb74fb510d849c5123d8e85def97ab3"
server_key_iv: "33ef9f970bbdfeb95178f33325c22157"
server_key_version: "encryptionkey@0"
I20260619 06:52:24.424592 2714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:24.452293 2714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:24.455744 2714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:24.455956 2714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:24.463214 2714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34237
I20260619 06:52:24.463241 2781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34237 every 8 connection(s)
I20260619 06:52:24.464164 2714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:24.465963 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 2714
I20260619 06:52:24.466113 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:24.466393 14803 external_mini_cluster.cc:1428] Setting key 269d659f3af263ef3817a4aff4d35099
I20260619 06:52:24.467031 2782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:24.473618 2782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659: Bootstrap starting.
I20260619 06:52:24.475142 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:24.467835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47491 (local address 127.0.0.1:34237)
0619 06:52:24.468239 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:24.468255 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:24.468288 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:52:24.468916 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:24.468932 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:24.469250 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:24.469514 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:24.469841 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:24.470777 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:24.471972 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:24.472420 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:24.472541 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:24.473461 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:24.473466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:24.473477 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:24.473902 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:24.473954 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:24.473976 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:24.474077 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:24.474313 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 06:52:24.474451 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":140,"threads_started":1}
I20260619 06:52:24.475955 2782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:24.476665 2782 log.cc:824] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:24.477945 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:24.478595 2782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659: No bootstrap required, opened a new log
I20260619 06:52:24.483027 2782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebea0305c4a4e678edf102bb0a1d659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } }
I20260619 06:52:24.483327 2782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:24.483393 2782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebea0305c4a4e678edf102bb0a1d659, State: Initialized, Role: FOLLOWER
I20260619 06:52:24.483887 2782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebea0305c4a4e678edf102bb0a1d659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } }
I20260619 06:52:24.484084 2782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:24.484198 2782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:24.484233 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:24.479928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40603 (local address 127.0.0.1:34237)
0619 06:52:24.480040 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:24.480043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:24.480245 (+ 202us) server_negotiation.cc:407] Connection header received
0619 06:52:24.480365 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:24.480367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:24.480398 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:24.480461 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:24.480789 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:24.481248 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:24.482387 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:24.482601 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:24.482675 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:24.483530 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:24.483533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:24.483536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:24.483548 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:24.483573 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:24.483587 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:24.483687 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:24.483982 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 06:52:24.484028 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:52:24.484339 2782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:24.485232 2782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebea0305c4a4e678edf102bb0a1d659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } }
I20260619 06:52:24.485559 2782 leader_election.cc:302] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ebea0305c4a4e678edf102bb0a1d659; no voters:
I20260619 06:52:24.485800 2782 leader_election.cc:288] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:24.485914 2792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:24.486156 2792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [term 1 LEADER]: Becoming Leader. State: Replica: 8ebea0305c4a4e678edf102bb0a1d659, State: Running, Role: LEADER
I20260619 06:52:24.486503 2792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebea0305c4a4e678edf102bb0a1d659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } }
I20260619 06:52:24.487136 2782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:24.488406 2794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ebea0305c4a4e678edf102bb0a1d659. Latest consensus state: current_term: 1 leader_uuid: "8ebea0305c4a4e678edf102bb0a1d659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebea0305c4a4e678edf102bb0a1d659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } } }
I20260619 06:52:24.488437 2793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ebea0305c4a4e678edf102bb0a1d659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ebea0305c4a4e678edf102bb0a1d659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34237 } } }
I20260619 06:52:24.488536 2794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:24.488536 2793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:24.488978 2802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:24.491706 2802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:24.496833 2802 catalog_manager.cc:1347] Generated new cluster ID: 71e32c6833f14169aa9ddea3249b4f92
I20260619 06:52:24.496901 2802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:24.524967 2802 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:24.526039 2802 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:24.542315 2802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ebea0305c4a4e678edf102bb0a1d659: Generated new TSK 0
I20260619 06:52:24.543018 2802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:24.550076 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 2714
W20260619 06:52:24.555192 2699 connection.cc:535] client connection to 127.0.0.1:34237 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34237): Connection reset by peer (error 104)
2026-06-19T06:52:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:24 dist-test-slave-9x3b krb5kdc[2813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:24 dist-test-slave-9x3b krb5kdc[2813](info): set up 2 sockets
Jun 19 06:52:24 dist-test-slave-9x3b krb5kdc[2813](info): commencing operation
krb5kdc: starting...
W20260619 06:52:25.579792 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:25 dist-test-slave-9x3b krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:25.696193 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:25.802938 2826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:25.803243 2826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:25.803333 2826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:25.807512 2826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:25.807611 2826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:25.807672 2826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:25.807730 2826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:25.807770 2826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:25.812989 2826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:25.814054 2826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:25.815138 2826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:25.823351 2834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:25.823414 2832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:25.823355 2831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:25.823627 2826 server_base.cc:1034] running on GCE node
I20260619 06:52:25.824239 2826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:25.825326 2826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:25.826517 2826 hybrid_clock.cc:648] HybridClock initialized: now 1781851945826504 us; error 26 us; skew 500 ppm
Jun 19 06:52:25 dist-test-slave-9x3b krb5kdc[2813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:25.829262 2826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:25.830158 2826 webserver.cc:460] Webserver started at http://127.0.0.1:42965/ using document root <none> and password file <none>
I20260619 06:52:25.830713 2826 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:25.830801 2826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:25.831037 2826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:25.833089 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "c49a8859cfaf49aebfd89ebb98e95663"
format_stamp: "Formatted at 2026-06-19 06:52:25 on dist-test-slave-9x3b"
server_key: "1bace5300b9d908908de4f478f074285"
server_key_iv: "e6a101e9a7bb8c8640137690620bbed3"
server_key_version: "encryptionkey@0"
I20260619 06:52:25.833658 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "c49a8859cfaf49aebfd89ebb98e95663"
format_stamp: "Formatted at 2026-06-19 06:52:25 on dist-test-slave-9x3b"
server_key: "1bace5300b9d908908de4f478f074285"
server_key_iv: "e6a101e9a7bb8c8640137690620bbed3"
server_key_version: "encryptionkey@0"
I20260619 06:52:25.837309 2826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:25.839706 2841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:25.840873 2826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:25.841017 2826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "c49a8859cfaf49aebfd89ebb98e95663"
format_stamp: "Formatted at 2026-06-19 06:52:25 on dist-test-slave-9x3b"
server_key: "1bace5300b9d908908de4f478f074285"
server_key_iv: "e6a101e9a7bb8c8640137690620bbed3"
server_key_version: "encryptionkey@0"
I20260619 06:52:25.841125 2826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:25.845582 2826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:25.848762 2826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:25.848955 2826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:25.855958 2826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36339
I20260619 06:52:25.855971 2893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36339 every 8 connection(s)
I20260619 06:52:25.856866 2826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:25.859436 2894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:25.862092 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 2826
I20260619 06:52:25.862238 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:25.862572 14803 external_mini_cluster.cc:1428] Setting key 3186cf1a21b7baa322f4656da52d68af
I20260619 06:52:25.865362 2894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663: Bootstrap starting.
I20260619 06:52:25.867519 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:25.863961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56639 (local address 127.0.0.1:36339)
0619 06:52:25.864393 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:25.864407 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:25.864440 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:52:25.865018 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:25.865034 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:25.865303 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:25.865664 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:25.866399 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:25.866403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:25.866414 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:25.866424 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:25.866474 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:25.866492 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:25.866633 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:25.866702 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 06:52:25.866854 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260619 06:52:25.867702 2894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:25.868474 2894 log.cc:824] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:25.870364 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:25.870489 2894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663: No bootstrap required, opened a new log
I20260619 06:52:25.874112 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:25.872499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52945 (local address 127.0.0.1:36339)
0619 06:52:25.872625 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:25.872630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:25.872804 (+ 174us) server_negotiation.cc:407] Connection header received
0619 06:52:25.872899 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:25.872902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:25.872936 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:25.873013 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:25.873689 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:25.873693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:25.873696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:25.873709 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:25.873739 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:25.873757 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:25.873850 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:25.873908 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 06:52:25.873952 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:52:25.875219 2894 raft_consensus.cc:348] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49a8859cfaf49aebfd89ebb98e95663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36339 } }
I20260619 06:52:25.875432 2894 raft_consensus.cc:374] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:25.875509 2894 raft_consensus.cc:729] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c49a8859cfaf49aebfd89ebb98e95663, State: Initialized, Role: FOLLOWER
I20260619 06:52:25.875934 2894 consensus_queue.cc:260] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49a8859cfaf49aebfd89ebb98e95663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36339 } }
I20260619 06:52:25.876073 2894 raft_consensus.cc:388] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:25.876138 2894 raft_consensus.cc:482] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:25.876264 2894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:25.877128 2894 raft_consensus.cc:504] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49a8859cfaf49aebfd89ebb98e95663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36339 } }
I20260619 06:52:25.877455 2894 leader_election.cc:302] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c49a8859cfaf49aebfd89ebb98e95663; no voters:
I20260619 06:52:25.877791 2894 leader_election.cc:288] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:25.877889 2904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:25.878090 2904 raft_consensus.cc:686] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [term 1 LEADER]: Becoming Leader. State: Replica: c49a8859cfaf49aebfd89ebb98e95663, State: Running, Role: LEADER
I20260619 06:52:25.878393 2904 consensus_queue.cc:237] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49a8859cfaf49aebfd89ebb98e95663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36339 } }
I20260619 06:52:25.878978 2894 sys_catalog.cc:564] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:25.879515 2904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c49a8859cfaf49aebfd89ebb98e95663. Latest consensus state: current_term: 1 leader_uuid: "c49a8859cfaf49aebfd89ebb98e95663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49a8859cfaf49aebfd89ebb98e95663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36339 } } }
I20260619 06:52:25.879704 2904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:25.879964 2905 sys_catalog.cc:455] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c49a8859cfaf49aebfd89ebb98e95663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49a8859cfaf49aebfd89ebb98e95663" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36339 } } }
I20260619 06:52:25.880117 2905 sys_catalog.cc:458] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:25.880105 2911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:25.883225 2911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:25.888859 2911 catalog_manager.cc:1347] Generated new cluster ID: e5b4b0ed901449feb5bb6d96517b0dd6
I20260619 06:52:25.888970 2911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:25.915066 2911 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:25.915427 2911 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:25.923941 2911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c49a8859cfaf49aebfd89ebb98e95663: Generated new TSK 0
I20260619 06:52:25.924633 2911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:25.932134 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 2826
2026-06-19T06:52:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:25 dist-test-slave-9x3b krb5kdc[2925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:25 dist-test-slave-9x3b krb5kdc[2925](info): set up 2 sockets
Jun 19 06:52:25 dist-test-slave-9x3b krb5kdc[2925](info): commencing operation
krb5kdc: starting...
W20260619 06:52:26.963802 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:27 dist-test-slave-9x3b krb5kdc[2925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:27.080396 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:27.186501 2938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:27.186745 2938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:27.186785 2938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:27.190773 2938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:27.190824 2938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:27.190843 2938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:27.190912 2938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:27.190948 2938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:27.196054 2938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:27.196977 2938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:27.197942 2938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:27.206002 2944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:27.206125 2938 server_base.cc:1034] running on GCE node
W20260619 06:52:27.206015 2946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:27.206008 2943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:27.206796 2938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:27.207851 2938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:27.209036 2938 hybrid_clock.cc:648] HybridClock initialized: now 1781851947209003 us; error 42 us; skew 500 ppm
Jun 19 06:52:27 dist-test-slave-9x3b krb5kdc[2925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:27.216487 2938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:27.217469 2938 webserver.cc:460] Webserver started at http://127.0.0.1:41765/ using document root <none> and password file <none>
I20260619 06:52:27.218016 2938 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:27.218077 2938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:27.218343 2938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:27.220470 2938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "e5307c9e36734c1a9b3676614414bf31"
format_stamp: "Formatted at 2026-06-19 06:52:27 on dist-test-slave-9x3b"
server_key: "0f3667ad096f614deb2cb64a11972c69"
server_key_iv: "44ee7a9f2e4154eeb83573a39760f8b3"
server_key_version: "encryptionkey@0"
I20260619 06:52:27.221045 2938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "e5307c9e36734c1a9b3676614414bf31"
format_stamp: "Formatted at 2026-06-19 06:52:27 on dist-test-slave-9x3b"
server_key: "0f3667ad096f614deb2cb64a11972c69"
server_key_iv: "44ee7a9f2e4154eeb83573a39760f8b3"
server_key_version: "encryptionkey@0"
I20260619 06:52:27.224690 2938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:52:27.227032 2953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:27.228263 2938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:27.228412 2938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "e5307c9e36734c1a9b3676614414bf31"
format_stamp: "Formatted at 2026-06-19 06:52:27 on dist-test-slave-9x3b"
server_key: "0f3667ad096f614deb2cb64a11972c69"
server_key_iv: "44ee7a9f2e4154eeb83573a39760f8b3"
server_key_version: "encryptionkey@0"
I20260619 06:52:27.228500 2938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:27.250547 2938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:27.253750 2938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:27.253955 2938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:27.260741 2938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41053
I20260619 06:52:27.260768 3005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41053 every 8 connection(s)
I20260619 06:52:27.261641 2938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:27.264225 3006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:27.266014 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 2938
I20260619 06:52:27.266157 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:27.266453 14803 external_mini_cluster.cc:1428] Setting key 251c4d8723454b67c1069c603bbd0643
I20260619 06:52:27.270388 3006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31: Bootstrap starting.
I20260619 06:52:27.272563 3006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:27.273248 3006 log.cc:824] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:27.275274 3006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31: No bootstrap required, opened a new log
Jun 19 06:52:27 dist-test-slave-9x3b krb5kdc[2925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:52:27.278434 3006 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5307c9e36734c1a9b3676614414bf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41053 } }
I20260619 06:52:27.278632 3006 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:27.278692 3006 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5307c9e36734c1a9b3676614414bf31, State: Initialized, Role: FOLLOWER
I20260619 06:52:27.279152 3006 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5307c9e36734c1a9b3676614414bf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41053 } }
I20260619 06:52:27.279273 3006 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:27.279352 3006 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:27.279461 3006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:27.280432 3006 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5307c9e36734c1a9b3676614414bf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41053 } }
I20260619 06:52:27.280758 3006 leader_election.cc:302] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5307c9e36734c1a9b3676614414bf31; no voters:
I20260619 06:52:27.280894 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:27.267876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47455 (local address 127.0.0.1:41053)
0619 06:52:27.268292 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:27.268307 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:27.268340 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:52:27.268958 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:27.268974 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:27.269281 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:27.269613 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:27.270422 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:27.271438 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:27.272364 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:27.272877 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:27.273039 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:27.275909 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:27.275914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:27.275926 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:27.275951 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:27.278059 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:27.278506 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:27.278511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:27.278515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:27.278643 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:27.278930 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:27.278933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:27.278935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:27.279384 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:27.279583 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:27.279845 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 06:52:27.280125 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260619 06:52:27.281090 3006 leader_election.cc:288] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:27.281215 3011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:27.281379 3011 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [term 1 LEADER]: Becoming Leader. State: Replica: e5307c9e36734c1a9b3676614414bf31, State: Running, Role: LEADER
I20260619 06:52:27.281711 3011 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5307c9e36734c1a9b3676614414bf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41053 } }
I20260619 06:52:27.282338 3006 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:27.283689 3011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5307c9e36734c1a9b3676614414bf31. Latest consensus state: current_term: 1 leader_uuid: "e5307c9e36734c1a9b3676614414bf31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5307c9e36734c1a9b3676614414bf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41053 } } }
I20260619 06:52:27.283864 3011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:27.284057 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:27.284093 3012 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5307c9e36734c1a9b3676614414bf31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5307c9e36734c1a9b3676614414bf31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41053 } } }
I20260619 06:52:27.284173 3012 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:27.284293 3023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:27.287734 3023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:27.293511 3023 catalog_manager.cc:1347] Generated new cluster ID: 65496dac4c454dfeac42a8830191d677
I20260619 06:52:27.293619 3023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:27.295015 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:27.286245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49195 (local address 127.0.0.1:41053)
0619 06:52:27.286345 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:27.286349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:27.286464 (+ 115us) server_negotiation.cc:407] Connection header received
0619 06:52:27.286627 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:27.286630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:27.286670 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:27.286715 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:27.287604 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:27.288384 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:27.289465 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:27.289813 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:27.289951 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:27.291522 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:27.291526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:27.291529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:27.291551 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:27.293084 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:27.293725 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:27.293729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:27.293731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:27.293819 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:27.294131 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:27.294134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:27.294136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:27.294363 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:27.294495 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:27.294733 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 06:52:27.294849 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:52:27.320545 3023 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:27.321501 3023 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:27.325726 3023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5307c9e36734c1a9b3676614414bf31: Generated new TSK 0
I20260619 06:52:27.326350 3023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:27.336061 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 2938
2026-06-19T06:52:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:27 dist-test-slave-9x3b krb5kdc[3037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:27 dist-test-slave-9x3b krb5kdc[3037](info): set up 2 sockets
Jun 19 06:52:27 dist-test-slave-9x3b krb5kdc[3037](info): commencing operation
krb5kdc: starting...
W20260619 06:52:28.363791 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:28 dist-test-slave-9x3b krb5kdc[3037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:28.477886 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:28.582921 3050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:28.583206 3050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:28.583283 3050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:28.587195 3050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:28.587301 3050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:28.587337 3050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:28.587426 3050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:28.587466 3050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:28.592991 3050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:28.594020 3050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:28.595019 3050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:28.603093 3055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:28.603084 3056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:28.603088 3058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:28.603276 3050 server_base.cc:1034] running on GCE node
I20260619 06:52:28.604003 3050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:28.604985 3050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:28.606141 3050 hybrid_clock.cc:648] HybridClock initialized: now 1781851948606128 us; error 24 us; skew 500 ppm
Jun 19 06:52:28 dist-test-slave-9x3b krb5kdc[3037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:28.608676 3050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:28.609478 3050 webserver.cc:460] Webserver started at http://127.0.0.1:44459/ using document root <none> and password file <none>
I20260619 06:52:28.610008 3050 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:28.610097 3050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:28.610319 3050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:28.612411 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "34df8a97eca44c3581646d7df771d514"
format_stamp: "Formatted at 2026-06-19 06:52:28 on dist-test-slave-9x3b"
server_key: "d2de300bbf6d36eca45d2cde5a5c2085"
server_key_iv: "863ef57538196fd5f6b71f4dcb18b5fa"
server_key_version: "encryptionkey@0"
I20260619 06:52:28.612977 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "34df8a97eca44c3581646d7df771d514"
format_stamp: "Formatted at 2026-06-19 06:52:28 on dist-test-slave-9x3b"
server_key: "d2de300bbf6d36eca45d2cde5a5c2085"
server_key_iv: "863ef57538196fd5f6b71f4dcb18b5fa"
server_key_version: "encryptionkey@0"
I20260619 06:52:28.616560 3050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:52:28.618772 3065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:28.619926 3050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:28.620066 3050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "34df8a97eca44c3581646d7df771d514"
format_stamp: "Formatted at 2026-06-19 06:52:28 on dist-test-slave-9x3b"
server_key: "d2de300bbf6d36eca45d2cde5a5c2085"
server_key_iv: "863ef57538196fd5f6b71f4dcb18b5fa"
server_key_version: "encryptionkey@0"
I20260619 06:52:28.620172 3050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:28.646595 3050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:28.649407 3050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:28.649607 3050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:28.656507 3050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45481
I20260619 06:52:28.656534 3117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45481 every 8 connection(s)
I20260619 06:52:28.657435 3050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:28.660094 3118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:28.663208 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 3050
I20260619 06:52:28.663358 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:28.663677 14803 external_mini_cluster.cc:1428] Setting key f8f41a2195471cc68e7706f470760aaf
I20260619 06:52:28.665992 3118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514: Bootstrap starting.
I20260619 06:52:28.668681 3118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:28.669519 3118 log.cc:824] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:28.671396 3118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514: No bootstrap required, opened a new log
I20260619 06:52:28.673337 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:28.665025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54895 (local address 127.0.0.1:45481)
0619 06:52:28.665458 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:28.665477 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:28.665517 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:52:28.666298 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:28.666319 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:28.666732 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:28.667074 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:28.667501 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:28.669031 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:28.669958 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:28.670424 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:28.670595 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:28.671395 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:28.671400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:28.671412 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:28.671880 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:28.671933 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:28.671956 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:28.672064 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:28.672206 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 06:52:28.672412 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
I20260619 06:52:28.674678 3118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34df8a97eca44c3581646d7df771d514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45481 } }
I20260619 06:52:28.674908 3118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:28.674999 3118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34df8a97eca44c3581646d7df771d514, State: Initialized, Role: FOLLOWER
I20260619 06:52:28.675592 3118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34df8a97eca44c3581646d7df771d514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45481 } }
I20260619 06:52:28.675770 3118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:28.675843 3118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:28.675945 3118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:28.676369 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:28.676941 3118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34df8a97eca44c3581646d7df771d514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45481 } }
I20260619 06:52:28.677265 3118 leader_election.cc:302] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34df8a97eca44c3581646d7df771d514; no voters:
I20260619 06:52:28.677574 3118 leader_election.cc:288] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:28.677632 3125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:28.677852 3125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [term 1 LEADER]: Becoming Leader. State: Replica: 34df8a97eca44c3581646d7df771d514, State: Running, Role: LEADER
I20260619 06:52:28.678748 3125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34df8a97eca44c3581646d7df771d514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45481 } }
I20260619 06:52:28.678872 3118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:28.680435 3129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34df8a97eca44c3581646d7df771d514. Latest consensus state: current_term: 1 leader_uuid: "34df8a97eca44c3581646d7df771d514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34df8a97eca44c3581646d7df771d514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45481 } } }
I20260619 06:52:28.680562 3129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:28.680810 3128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34df8a97eca44c3581646d7df771d514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34df8a97eca44c3581646d7df771d514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45481 } } }
I20260619 06:52:28.680982 3128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:28.683151 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:28.678628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42077 (local address 127.0.0.1:45481)
0619 06:52:28.678745 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:28.678752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:28.678767 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:52:28.678801 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:28.678804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:28.678841 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:28.678909 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:28.679248 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:28.679970 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:28.681317 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:28.681654 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:28.682117 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:28.682543 (+ 426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:28.682546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:28.682549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:28.682563 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:28.682593 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:28.682605 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:28.682939 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:28.682973 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 06:52:28.683023 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 06:52:28.683521 3143 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:28.683597 3143 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:28.683682 3142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:28.684528 3142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:28.689443 3142 catalog_manager.cc:1347] Generated new cluster ID: 9a53eb6bcabc43c39ea2066ed20a7a54
I20260619 06:52:28.689567 3142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:28.721395 3142 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:28.722435 3142 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:28.732757 3142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34df8a97eca44c3581646d7df771d514: Generated new TSK 0
I20260619 06:52:28.733568 3142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:28.737116 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 3050
2026-06-19T06:52:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:28 dist-test-slave-9x3b krb5kdc[3149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:28 dist-test-slave-9x3b krb5kdc[3149](info): set up 2 sockets
Jun 19 06:52:28 dist-test-slave-9x3b krb5kdc[3149](info): commencing operation
krb5kdc: starting...
W20260619 06:52:29.771807 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:29 dist-test-slave-9x3b krb5kdc[3149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:29.888198 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:29.995693 3162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:29.995927 3162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:29.996006 3162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:29.999935 3162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:29.999991 3162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:30.000011 3162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:30.000057 3162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:30.000092 3162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:30.005450 3162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:30.006644 3162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:30.007781 3162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:30.016470 3170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:30.016664 3162 server_base.cc:1034] running on GCE node
W20260619 06:52:30.016494 3167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:30.016464 3168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:30.017422 3162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:30.018683 3162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:30.019867 3162 hybrid_clock.cc:648] HybridClock initialized: now 1781851950019834 us; error 54 us; skew 500 ppm
Jun 19 06:52:30 dist-test-slave-9x3b krb5kdc[3149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:30.022749 3162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:30.023773 3162 webserver.cc:460] Webserver started at http://127.0.0.1:42941/ using document root <none> and password file <none>
I20260619 06:52:30.024300 3162 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:30.024358 3162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:30.024537 3162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:30.026510 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "5bf26caf22b0496b8f982c23c6857901"
format_stamp: "Formatted at 2026-06-19 06:52:30 on dist-test-slave-9x3b"
server_key: "96b718dcf9191a2466f432c928495a62"
server_key_iv: "9836159bf57e149cb7b7e0dc385b0fe1"
server_key_version: "encryptionkey@0"
I20260619 06:52:30.027004 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "5bf26caf22b0496b8f982c23c6857901"
format_stamp: "Formatted at 2026-06-19 06:52:30 on dist-test-slave-9x3b"
server_key: "96b718dcf9191a2466f432c928495a62"
server_key_iv: "9836159bf57e149cb7b7e0dc385b0fe1"
server_key_version: "encryptionkey@0"
I20260619 06:52:30.030623 3162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:52:30.033000 3177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:30.034200 3162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:52:30.034365 3162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "5bf26caf22b0496b8f982c23c6857901"
format_stamp: "Formatted at 2026-06-19 06:52:30 on dist-test-slave-9x3b"
server_key: "96b718dcf9191a2466f432c928495a62"
server_key_iv: "9836159bf57e149cb7b7e0dc385b0fe1"
server_key_version: "encryptionkey@0"
I20260619 06:52:30.034472 3162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:30.039120 3162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:30.042335 3162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:30.042534 3162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:30.049655 3162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42329
I20260619 06:52:30.049675 3229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42329 every 8 connection(s)
I20260619 06:52:30.050626 3162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:30.053365 3230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:30.053645 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 3162
I20260619 06:52:30.053794 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:30.054093 14803 external_mini_cluster.cc:1428] Setting key bc9d32f6d333300e4cde18e302637048
I20260619 06:52:30.060119 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:30.055571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41041 (local address 127.0.0.1:42329)
0619 06:52:30.056080 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:30.056101 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:30.056145 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:52:30.056933 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:30.056959 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:30.057341 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:30.057722 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:30.058592 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:30.058598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:30.058614 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:30.058626 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:30.058693 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:30.058720 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:30.058871 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:30.058956 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 06:52:30.059118 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":114,"threads_started":1}
I20260619 06:52:30.061067 3230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901: Bootstrap starting.
I20260619 06:52:30.063107 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:30.063678 3230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:30.064458 3230 log.cc:824] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:30.066641 3230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901: No bootstrap required, opened a new log
I20260619 06:52:30.067157 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:30.065283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54945 (local address 127.0.0.1:42329)
0619 06:52:30.065407 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:30.065412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:30.065629 (+ 217us) server_negotiation.cc:407] Connection header received
0619 06:52:30.065727 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:30.065730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:30.065765 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:30.065856 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:30.066668 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:30.066672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:30.066677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:30.066692 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:30.066719 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:30.066733 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:30.066815 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:30.066969 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 06:52:30.067006 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:52:30.070511 3230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf26caf22b0496b8f982c23c6857901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260619 06:52:30.070741 3230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:30.070822 3230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf26caf22b0496b8f982c23c6857901, State: Initialized, Role: FOLLOWER
I20260619 06:52:30.071271 3230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf26caf22b0496b8f982c23c6857901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260619 06:52:30.071421 3230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:30.071511 3230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:30.071647 3230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:30.072611 3230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf26caf22b0496b8f982c23c6857901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260619 06:52:30.072966 3230 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf26caf22b0496b8f982c23c6857901; no voters:
I20260619 06:52:30.073310 3230 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:30.073467 3240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:30.073717 3240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [term 1 LEADER]: Becoming Leader. State: Replica: 5bf26caf22b0496b8f982c23c6857901, State: Running, Role: LEADER
I20260619 06:52:30.074106 3240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf26caf22b0496b8f982c23c6857901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260619 06:52:30.074668 3230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:30.076090 3242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf26caf22b0496b8f982c23c6857901. Latest consensus state: current_term: 1 leader_uuid: "5bf26caf22b0496b8f982c23c6857901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf26caf22b0496b8f982c23c6857901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260619 06:52:30.076125 3241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf26caf22b0496b8f982c23c6857901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf26caf22b0496b8f982c23c6857901" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260619 06:52:30.076208 3242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:30.076237 3241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:30.076610 3251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:30.079429 3251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:30.084898 3251 catalog_manager.cc:1347] Generated new cluster ID: 3e12ab2860b1473faa6feeb329db14dc
I20260619 06:52:30.084975 3251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:30.097754 3251 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:30.098111 3251 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:30.107213 3251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bf26caf22b0496b8f982c23c6857901: Generated new TSK 0
I20260619 06:52:30.108095 3251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:30.114951 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 3162
2026-06-19T06:52:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:30 dist-test-slave-9x3b krb5kdc[3261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:30 dist-test-slave-9x3b krb5kdc[3261](info): set up 2 sockets
Jun 19 06:52:30 dist-test-slave-9x3b krb5kdc[3261](info): commencing operation
krb5kdc: starting...
W20260619 06:52:31.151773 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:31 dist-test-slave-9x3b krb5kdc[3261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:31.278923 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:31.385344 3274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:31.385586 3274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:31.385625 3274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:31.389613 3274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:31.389669 3274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:31.389689 3274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:31.389705 3274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:31.389760 3274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:31.394865 3274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:31.395877 3274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:31.396823 3274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:31.404852 3280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:31.404834 3282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:31.404860 3279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:31.405165 3274 server_base.cc:1034] running on GCE node
I20260619 06:52:31.405687 3274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:31.406659 3274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:31.407804 3274 hybrid_clock.cc:648] HybridClock initialized: now 1781851951407778 us; error 39 us; skew 500 ppm
Jun 19 06:52:31 dist-test-slave-9x3b krb5kdc[3261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:31.410416 3274 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:31.411250 3274 webserver.cc:460] Webserver started at http://127.0.0.1:36421/ using document root <none> and password file <none>
I20260619 06:52:31.411821 3274 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:31.411916 3274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:31.412132 3274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:31.414022 3274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "57ca8aa202204727a71fe1374eedfbd6"
format_stamp: "Formatted at 2026-06-19 06:52:31 on dist-test-slave-9x3b"
server_key: "e08935484379727751037432f3749f5b"
server_key_iv: "31ecb249509cec0998213b38394d89f1"
server_key_version: "encryptionkey@0"
I20260619 06:52:31.414561 3274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "57ca8aa202204727a71fe1374eedfbd6"
format_stamp: "Formatted at 2026-06-19 06:52:31 on dist-test-slave-9x3b"
server_key: "e08935484379727751037432f3749f5b"
server_key_iv: "31ecb249509cec0998213b38394d89f1"
server_key_version: "encryptionkey@0"
I20260619 06:52:31.417969 3274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:31.420187 3289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:31.421298 3274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:31.421439 3274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "57ca8aa202204727a71fe1374eedfbd6"
format_stamp: "Formatted at 2026-06-19 06:52:31 on dist-test-slave-9x3b"
server_key: "e08935484379727751037432f3749f5b"
server_key_iv: "31ecb249509cec0998213b38394d89f1"
server_key_version: "encryptionkey@0"
I20260619 06:52:31.421545 3274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:31.434450 3274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:31.437490 3274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:31.437673 3274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:31.443989 3274 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40723
I20260619 06:52:31.444018 3341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40723 every 8 connection(s)
I20260619 06:52:31.444865 3274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:31.447438 3342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:31.452729 3342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6: Bootstrap starting.
I20260619 06:52:31.454592 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 3274
I20260619 06:52:31.454742 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:31.455035 14803 external_mini_cluster.cc:1428] Setting key caa31f626953585d7b295e18d95eb571
I20260619 06:52:31.455319 3342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:31.456244 3342 log.cc:824] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:31.458220 3342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6: No bootstrap required, opened a new log
I20260619 06:52:31.462450 3342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ca8aa202204727a71fe1374eedfbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } }
I20260619 06:52:31.462749 3342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:31.462826 3342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ca8aa202204727a71fe1374eedfbd6, State: Initialized, Role: FOLLOWER
I20260619 06:52:31.463296 3342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ca8aa202204727a71fe1374eedfbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } }
I20260619 06:52:31.463423 3342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:31.463506 3342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:31.463632 3342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 06:52:31 dist-test-slave-9x3b krb5kdc[3261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:52:31.464691 3342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ca8aa202204727a71fe1374eedfbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } }
I20260619 06:52:31.465027 3342 leader_election.cc:302] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57ca8aa202204727a71fe1374eedfbd6; no voters:
I20260619 06:52:31.465317 3342 leader_election.cc:288] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:31.465528 3347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:31.465741 3347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [term 1 LEADER]: Becoming Leader. State: Replica: 57ca8aa202204727a71fe1374eedfbd6, State: Running, Role: LEADER
I20260619 06:52:31.466049 3347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ca8aa202204727a71fe1374eedfbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } }
I20260619 06:52:31.466540 3342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:31.467828 3348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57ca8aa202204727a71fe1374eedfbd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ca8aa202204727a71fe1374eedfbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } } }
I20260619 06:52:31.467958 3348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:31.468173 3349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57ca8aa202204727a71fe1374eedfbd6. Latest consensus state: current_term: 1 leader_uuid: "57ca8aa202204727a71fe1374eedfbd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ca8aa202204727a71fe1374eedfbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } } }
I20260619 06:52:31.468286 3349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:31.470220 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:31.456123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:40011 (local address 127.0.0.1:40723)
0619 06:52:31.456513 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:31.456538 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:31.456571 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:52:31.457178 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:31.457193 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:31.457510 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:31.457778 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:31.458695 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:31.459671 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:31.460870 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:31.461325 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:31.461436 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:31.464380 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:31.464385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:31.464397 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:31.464422 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:31.466399 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:31.467032 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:31.467039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:31.467044 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:31.467184 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:31.467644 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:31.467649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:31.467650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:31.468164 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:31.468383 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:31.469117 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 06:52:31.469337 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":139,"threads_started":1}
W20260619 06:52:31.471844 3362 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:31.471922 3362 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:31.472033 3357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:31.472909 3357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:31.473276 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:31.478456 3357 catalog_manager.cc:1347] Generated new cluster ID: f487175f38bd492b8e4d0fba3fdc2514
I20260619 06:52:31.478525 3357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:31.484999 3345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:31.475370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:52403 (local address 127.0.0.1:40723)
0619 06:52:31.475482 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:31.475486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:31.475906 (+ 420us) server_negotiation.cc:407] Connection header received
0619 06:52:31.476023 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:31.476026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:31.476072 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:31.476154 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:31.476847 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:31.478059 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:31.479318 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:31.479734 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:31.479810 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:31.481568 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:31.481571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:31.481574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:31.481598 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:31.483195 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:31.483725 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:31.483728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:31.483730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:31.483813 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:31.484146 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:31.484149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:31.484151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:31.484355 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:31.484492 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:31.484754 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 06:52:31.484845 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:52:31.507431 3357 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:31.508708 3357 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:31.520301 3357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57ca8aa202204727a71fe1374eedfbd6: Generated new TSK 0
I20260619 06:52:31.521090 3357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:31.527431 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 3274
W20260619 06:52:31.532943 3259 connection.cc:535] client connection to 127.0.0.1:40723 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40723): Connection reset by peer (error 104)
2026-06-19T06:52:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:31 dist-test-slave-9x3b krb5kdc[3373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:31 dist-test-slave-9x3b krb5kdc[3373](info): set up 2 sockets
Jun 19 06:52:31 dist-test-slave-9x3b krb5kdc[3373](info): commencing operation
krb5kdc: starting...
W20260619 06:52:32.571828 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:32 dist-test-slave-9x3b krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:32.686121 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:32.791368 3386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:32.791640 3386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:32.791682 3386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:32.795910 3386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:32.795980 3386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:32.796002 3386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:32.796018 3386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:32.796034 3386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:32.801183 3386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:32.802191 3386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:32.803200 3386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:32.811323 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:32.811323 3391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:32.811326 3392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:32.811749 3386 server_base.cc:1034] running on GCE node
I20260619 06:52:32.812273 3386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:32.813364 3386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:32.814556 3386 hybrid_clock.cc:648] HybridClock initialized: now 1781851952814548 us; error 28 us; skew 500 ppm
Jun 19 06:52:32 dist-test-slave-9x3b krb5kdc[3373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:32.821398 3386 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:32.822190 3386 webserver.cc:460] Webserver started at http://127.0.0.1:44611/ using document root <none> and password file <none>
I20260619 06:52:32.822690 3386 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:32.822747 3386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:32.822914 3386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:32.824877 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "d717697151ee4314bd0c153b6c17fd69"
format_stamp: "Formatted at 2026-06-19 06:52:32 on dist-test-slave-9x3b"
server_key: "471f1cfea8a3c39b2fec5061e14b414a"
server_key_iv: "3e589786dde4ac6eb6f9cc63ca73834e"
server_key_version: "encryptionkey@0"
I20260619 06:52:32.825377 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "d717697151ee4314bd0c153b6c17fd69"
format_stamp: "Formatted at 2026-06-19 06:52:32 on dist-test-slave-9x3b"
server_key: "471f1cfea8a3c39b2fec5061e14b414a"
server_key_iv: "3e589786dde4ac6eb6f9cc63ca73834e"
server_key_version: "encryptionkey@0"
I20260619 06:52:32.828792 3386 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:52:32.830943 3401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:32.832147 3386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:32.832249 3386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "d717697151ee4314bd0c153b6c17fd69"
format_stamp: "Formatted at 2026-06-19 06:52:32 on dist-test-slave-9x3b"
server_key: "471f1cfea8a3c39b2fec5061e14b414a"
server_key_iv: "3e589786dde4ac6eb6f9cc63ca73834e"
server_key_version: "encryptionkey@0"
I20260619 06:52:32.832336 3386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:32.848441 3386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:32.851037 3386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:32.851243 3386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:32.858109 3386 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40977
I20260619 06:52:32.858126 3453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40977 every 8 connection(s)
I20260619 06:52:32.859014 3386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:32.861555 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 3386
I20260619 06:52:32.861724 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:32.861891 3454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:32.862049 14803 external_mini_cluster.cc:1428] Setting key 6d3536d48289e9b105c67a4bcb616b60
I20260619 06:52:32.869150 3454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69: Bootstrap starting.
I20260619 06:52:32.871829 3454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:32.872102 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:32.863346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:59311 (local address 127.0.0.1:40977)
0619 06:52:32.863787 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:32.863809 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:32.863851 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:52:32.864739 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:32.864762 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:32.865197 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:32.865568 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:32.865970 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:32.867405 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:32.868414 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:32.869081 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:32.869242 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:32.870146 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:32.870152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:32.870168 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:32.870679 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:32.870743 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:32.870771 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:32.870898 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:32.871024 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 06:52:32.871147 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":114,"threads_started":1}
I20260619 06:52:32.872627 3454 log.cc:824] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:32.874645 3454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69: No bootstrap required, opened a new log
I20260619 06:52:32.875028 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:32.878770 3454 raft_consensus.cc:348] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717697151ee4314bd0c153b6c17fd69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260619 06:52:32.879007 3454 raft_consensus.cc:374] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:32.879093 3454 raft_consensus.cc:729] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d717697151ee4314bd0c153b6c17fd69, State: Initialized, Role: FOLLOWER
I20260619 06:52:32.879524 3454 consensus_queue.cc:260] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717697151ee4314bd0c153b6c17fd69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260619 06:52:32.879717 3454 raft_consensus.cc:388] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:32.879779 3454 raft_consensus.cc:482] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:32.879871 3454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:32.881098 3454 raft_consensus.cc:504] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717697151ee4314bd0c153b6c17fd69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260619 06:52:32.881258 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:32.877086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:37243 (local address 127.0.0.1:40977)
0619 06:52:32.877208 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:32.877212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:32.877341 (+ 129us) server_negotiation.cc:407] Connection header received
0619 06:52:32.877455 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:32.877458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:32.877511 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:32.877597 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:32.877827 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:32.878563 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:32.879596 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:32.879962 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:32.880081 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:32.880828 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:32.880832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:32.880834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:32.880847 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:32.880874 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:32.880889 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:32.880993 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:32.881087 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 06:52:32.881121 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:52:32.881484 3454 leader_election.cc:302] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d717697151ee4314bd0c153b6c17fd69; no voters:
I20260619 06:52:32.881798 3454 leader_election.cc:288] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:32.881882 3464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:32.882107 3464 raft_consensus.cc:686] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [term 1 LEADER]: Becoming Leader. State: Replica: d717697151ee4314bd0c153b6c17fd69, State: Running, Role: LEADER
I20260619 06:52:32.882457 3464 consensus_queue.cc:237] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717697151ee4314bd0c153b6c17fd69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260619 06:52:32.882975 3454 sys_catalog.cc:564] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:32.883971 3466 sys_catalog.cc:455] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d717697151ee4314bd0c153b6c17fd69. Latest consensus state: current_term: 1 leader_uuid: "d717697151ee4314bd0c153b6c17fd69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717697151ee4314bd0c153b6c17fd69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260619 06:52:32.884081 3466 sys_catalog.cc:458] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:32.884377 3465 sys_catalog.cc:455] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d717697151ee4314bd0c153b6c17fd69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d717697151ee4314bd0c153b6c17fd69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260619 06:52:32.884459 3465 sys_catalog.cc:458] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69 [sys.catalog]: This master's current role is: LEADER
W20260619 06:52:32.887454 3479 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:32.887542 3479 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:32.887638 3473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:32.888298 3473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:32.893963 3473 catalog_manager.cc:1347] Generated new cluster ID: 91ce31568a0f45d98be60135fc0ef6e3
I20260619 06:52:32.894035 3473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:32.930722 3473 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:32.931895 3473 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:32.940749 3473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d717697151ee4314bd0c153b6c17fd69: Generated new TSK 0
I20260619 06:52:32.941454 3473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:32.959287 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 3386
W20260619 06:52:32.963932 3371 connection.cc:535] client connection to 127.0.0.1:40977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40977): Connection reset by peer (error 104)
2026-06-19T06:52:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:33 dist-test-slave-9x3b krb5kdc[3485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:33 dist-test-slave-9x3b krb5kdc[3485](info): set up 2 sockets
Jun 19 06:52:33 dist-test-slave-9x3b krb5kdc[3485](info): commencing operation
krb5kdc: starting...
W20260619 06:52:33.995908 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:34 dist-test-slave-9x3b krb5kdc[3485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:34.110792 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:34.219653 3498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:34.219945 3498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:34.220018 3498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:34.224012 3498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:34.224090 3498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:34.224128 3498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:34.224182 3498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:34.224232 3498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:34.229456 3498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:34.230518 3498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:34.231515 3498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:34.239621 3503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:34.239663 3506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:34.239621 3504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:34.239981 3498 server_base.cc:1034] running on GCE node
I20260619 06:52:34.240546 3498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:34.241606 3498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:34.242795 3498 hybrid_clock.cc:648] HybridClock initialized: now 1781851954242785 us; error 26 us; skew 500 ppm
Jun 19 06:52:34 dist-test-slave-9x3b krb5kdc[3485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:34.245535 3498 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:34.246417 3498 webserver.cc:460] Webserver started at http://127.0.0.1:46837/ using document root <none> and password file <none>
I20260619 06:52:34.246949 3498 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:34.247006 3498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:34.247164 3498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:34.249135 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "12deeaf834d144ab9427e212c5b5a166"
format_stamp: "Formatted at 2026-06-19 06:52:34 on dist-test-slave-9x3b"
server_key: "a668455128b4fd5e8682dd1b1493f3a9"
server_key_iv: "876257c3922ca0e17a6294f9a4855d76"
server_key_version: "encryptionkey@0"
I20260619 06:52:34.249621 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "12deeaf834d144ab9427e212c5b5a166"
format_stamp: "Formatted at 2026-06-19 06:52:34 on dist-test-slave-9x3b"
server_key: "a668455128b4fd5e8682dd1b1493f3a9"
server_key_iv: "876257c3922ca0e17a6294f9a4855d76"
server_key_version: "encryptionkey@0"
I20260619 06:52:34.253185 3498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 06:52:34.255445 3513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:34.256708 3498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 06:52:34.256836 3498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "12deeaf834d144ab9427e212c5b5a166"
format_stamp: "Formatted at 2026-06-19 06:52:34 on dist-test-slave-9x3b"
server_key: "a668455128b4fd5e8682dd1b1493f3a9"
server_key_iv: "876257c3922ca0e17a6294f9a4855d76"
server_key_version: "encryptionkey@0"
I20260619 06:52:34.256945 3498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:34.261415 3498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:34.264596 3498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:34.264791 3498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:34.271492 3498 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33455
I20260619 06:52:34.271514 3565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33455 every 8 connection(s)
I20260619 06:52:34.272400 3498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:34.275059 3566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:34.276639 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 3498
I20260619 06:52:34.276785 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:34.277072 14803 external_mini_cluster.cc:1428] Setting key 8c426f7b029ed774aca8f7313eb9d983
I20260619 06:52:34.281478 3566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166: Bootstrap starting.
I20260619 06:52:34.282027 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:34.278355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:60043 (local address 127.0.0.1:33455)
0619 06:52:34.278750 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:34.278765 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:34.278799 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:52:34.279403 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:34.279420 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:34.279706 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:34.279979 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:34.280784 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:34.280789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:34.280799 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:34.280809 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:34.280858 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:34.280877 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:34.280994 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:34.281093 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 06:52:34.281256 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
I20260619 06:52:34.283850 3566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:34.284634 3566 log.cc:824] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:34.284794 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:34.286531 3566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166: No bootstrap required, opened a new log
I20260619 06:52:34.288640 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:34.286894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:38299 (local address 127.0.0.1:33455)
0619 06:52:34.287009 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:34.287013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:34.287346 (+ 333us) server_negotiation.cc:407] Connection header received
0619 06:52:34.287434 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:34.287437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:34.287469 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:34.287555 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:34.288277 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:34.288281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:34.288284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:34.288295 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:34.288325 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:34.288337 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:34.288431 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:34.288479 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 06:52:34.288525 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:52:34.290644 3566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12deeaf834d144ab9427e212c5b5a166" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } }
I20260619 06:52:34.290854 3566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:34.290886 3566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12deeaf834d144ab9427e212c5b5a166, State: Initialized, Role: FOLLOWER
I20260619 06:52:34.291232 3566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12deeaf834d144ab9427e212c5b5a166" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } }
I20260619 06:52:34.291337 3566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:34.291379 3566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:34.291443 3566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:34.292291 3566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12deeaf834d144ab9427e212c5b5a166" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } }
I20260619 06:52:34.292577 3566 leader_election.cc:302] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12deeaf834d144ab9427e212c5b5a166; no voters:
I20260619 06:52:34.292855 3566 leader_election.cc:288] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:34.292971 3576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:34.293206 3576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [term 1 LEADER]: Becoming Leader. State: Replica: 12deeaf834d144ab9427e212c5b5a166, State: Running, Role: LEADER
I20260619 06:52:34.293514 3576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12deeaf834d144ab9427e212c5b5a166" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } }
I20260619 06:52:34.294122 3566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:34.295131 3577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12deeaf834d144ab9427e212c5b5a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12deeaf834d144ab9427e212c5b5a166" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } } }
I20260619 06:52:34.295251 3577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:34.295544 3578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12deeaf834d144ab9427e212c5b5a166. Latest consensus state: current_term: 1 leader_uuid: "12deeaf834d144ab9427e212c5b5a166" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12deeaf834d144ab9427e212c5b5a166" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33455 } } }
I20260619 06:52:34.295652 3578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:34.295764 3585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:34.298717 3585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:34.304154 3585 catalog_manager.cc:1347] Generated new cluster ID: c02b6e198209454fa0641e0fd6da5dca
I20260619 06:52:34.304215 3585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:34.314112 3585 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:34.314425 3585 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:34.319013 3585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12deeaf834d144ab9427e212c5b5a166: Generated new TSK 0
I20260619 06:52:34.319769 3585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:34.324270 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 3498
2026-06-19T06:52:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:34 dist-test-slave-9x3b krb5kdc[3597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:34 dist-test-slave-9x3b krb5kdc[3597](info): set up 2 sockets
Jun 19 06:52:34 dist-test-slave-9x3b krb5kdc[3597](info): commencing operation
krb5kdc: starting...
W20260619 06:52:35.363967 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:35 dist-test-slave-9x3b krb5kdc[3597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:35.478317 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:35.586328 3610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:35.586652 3610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:35.586736 3610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:35.590878 3610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:35.590972 3610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:35.591012 3610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:35.591070 3610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:35.591110 3610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:35.596477 3610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:35.597553 3610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:35.598587 3610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:35.607152 3618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:35.607333 3615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:35.607153 3616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:35.607576 3610 server_base.cc:1034] running on GCE node
I20260619 06:52:35.608209 3610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:35.609443 3610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:35.610637 3610 hybrid_clock.cc:648] HybridClock initialized: now 1781851955610623 us; error 29 us; skew 500 ppm
Jun 19 06:52:35 dist-test-slave-9x3b krb5kdc[3597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:35.613472 3610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:35.614459 3610 webserver.cc:460] Webserver started at http://127.0.0.1:36671/ using document root <none> and password file <none>
I20260619 06:52:35.615031 3610 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:35.615123 3610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:35.615355 3610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:35.617451 3610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "18c037f06e9b40fe998f6a9a03330855"
format_stamp: "Formatted at 2026-06-19 06:52:35 on dist-test-slave-9x3b"
server_key: "903bbe36421d3e968800d55d1a8b96ea"
server_key_iv: "e365ff22aebda2d69e411a062f5071b5"
server_key_version: "encryptionkey@0"
I20260619 06:52:35.618026 3610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "18c037f06e9b40fe998f6a9a03330855"
format_stamp: "Formatted at 2026-06-19 06:52:35 on dist-test-slave-9x3b"
server_key: "903bbe36421d3e968800d55d1a8b96ea"
server_key_iv: "e365ff22aebda2d69e411a062f5071b5"
server_key_version: "encryptionkey@0"
I20260619 06:52:35.621809 3610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:52:35.624189 3625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:35.625407 3610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:35.625555 3610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "18c037f06e9b40fe998f6a9a03330855"
format_stamp: "Formatted at 2026-06-19 06:52:35 on dist-test-slave-9x3b"
server_key: "903bbe36421d3e968800d55d1a8b96ea"
server_key_iv: "e365ff22aebda2d69e411a062f5071b5"
server_key_version: "encryptionkey@0"
I20260619 06:52:35.625670 3610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:35.638242 3610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:35.641453 3610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:35.641654 3610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:35.648689 3610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40097
I20260619 06:52:35.648712 3677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40097 every 8 connection(s)
I20260619 06:52:35.649634 3610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:35.652333 3678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:35.654500 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 3610
I20260619 06:52:35.654641 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:35.654937 14803 external_mini_cluster.cc:1428] Setting key ba11941c683714bca22aff7730a1bcc0
I20260619 06:52:35.658402 3678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855: Bootstrap starting.
I20260619 06:52:35.660562 3678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:35.661268 3678 log.cc:824] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:35.663384 3678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855: No bootstrap required, opened a new log
Jun 19 06:52:35 dist-test-slave-9x3b krb5kdc[3597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:52:35.667026 3678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c037f06e9b40fe998f6a9a03330855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260619 06:52:35.667292 3678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:35.667364 3678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c037f06e9b40fe998f6a9a03330855, State: Initialized, Role: FOLLOWER
I20260619 06:52:35.667860 3678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c037f06e9b40fe998f6a9a03330855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260619 06:52:35.668046 3678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:35.668124 3678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:35.668234 3678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:35.669293 3678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c037f06e9b40fe998f6a9a03330855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260619 06:52:35.669651 3678 leader_election.cc:302] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18c037f06e9b40fe998f6a9a03330855; no voters:
I20260619 06:52:35.669946 3678 leader_election.cc:288] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:35.670056 3683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:35.670253 3683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [term 1 LEADER]: Becoming Leader. State: Replica: 18c037f06e9b40fe998f6a9a03330855, State: Running, Role: LEADER
I20260619 06:52:35.670324 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:35.656323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:35663 (local address 127.0.0.1:40097)
0619 06:52:35.656715 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:35.656729 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:35.656763 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:52:35.657583 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:35.657601 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:35.657935 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:35.658260 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:35.659042 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:35.660092 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:35.660995 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:35.661499 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:35.661662 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:35.664943 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:35.664948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:35.664960 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:35.664985 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:35.667098 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:35.667702 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:35.667707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:35.667713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:35.667828 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:35.668204 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:35.668209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:35.668211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:35.668642 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:35.668804 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:35.669149 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 06:52:35.669373 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260619 06:52:35.670549 3683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c037f06e9b40fe998f6a9a03330855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260619 06:52:35.671247 3678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:35.671654 3685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18c037f06e9b40fe998f6a9a03330855. Latest consensus state: current_term: 1 leader_uuid: "18c037f06e9b40fe998f6a9a03330855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c037f06e9b40fe998f6a9a03330855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } } }
I20260619 06:52:35.671759 3685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:35.672729 3684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18c037f06e9b40fe998f6a9a03330855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c037f06e9b40fe998f6a9a03330855" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } } }
I20260619 06:52:35.672899 3684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:35.673980 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 06:52:35.676290 3699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:35.676373 3699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:35.676446 3687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:35.677412 3687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:35.683346 3687 catalog_manager.cc:1347] Generated new cluster ID: ba83e99251ae4c938cbddac74c4f2fb0
I20260619 06:52:35.683413 3687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:35.684739 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:35.676022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:38311 (local address 127.0.0.1:40097)
0619 06:52:35.676121 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:35.676125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:35.676374 (+ 249us) server_negotiation.cc:407] Connection header received
0619 06:52:35.676606 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:35.676609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:35.676658 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:35.676746 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:35.677592 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:35.678365 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:35.679403 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:35.679698 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:35.679811 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:35.681196 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:35.681199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:35.681202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:35.681228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:35.682747 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:35.683233 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:35.683237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:35.683240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:35.683326 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:35.683791 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:35.683795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:35.683797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:35.684020 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:35.684174 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:35.684483 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 06:52:35.684583 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:52:35.696182 3687 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:35.697501 3687 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:35.703032 3687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18c037f06e9b40fe998f6a9a03330855: Generated new TSK 0
I20260619 06:52:35.703886 3687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:35.712078 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 3610
W20260619 06:52:35.717422 3595 connection.cc:535] client connection to 127.0.0.1:40097 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40097): Connection reset by peer (error 104)
2026-06-19T06:52:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:35 dist-test-slave-9x3b krb5kdc[3709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:35 dist-test-slave-9x3b krb5kdc[3709](info): set up 2 sockets
Jun 19 06:52:35 dist-test-slave-9x3b krb5kdc[3709](info): commencing operation
krb5kdc: starting...
W20260619 06:52:36.743952 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:36 dist-test-slave-9x3b krb5kdc[3709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:36.861862 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:36.968832 3722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:36.969130 3722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:36.969211 3722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:36.973176 3722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:36.973269 3722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:36.973306 3722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:36.973366 3722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:36.973405 3722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:36.978571 3722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:36.979581 3722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:36.980580 3722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:36.988569 3728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:36.988580 3727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:36.988579 3730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:36.988842 3722 server_base.cc:1034] running on GCE node
I20260619 06:52:36.989377 3722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:36.990401 3722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:36.991586 3722 hybrid_clock.cc:648] HybridClock initialized: now 1781851956991573 us; error 26 us; skew 500 ppm
Jun 19 06:52:36 dist-test-slave-9x3b krb5kdc[3709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:36.994287 3722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:36.995151 3722 webserver.cc:460] Webserver started at http://127.0.0.1:42489/ using document root <none> and password file <none>
I20260619 06:52:36.995751 3722 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:36.995844 3722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:36.996075 3722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:36.998049 3722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "333e548bbc634330a7f84ca932202180"
format_stamp: "Formatted at 2026-06-19 06:52:36 on dist-test-slave-9x3b"
server_key: "ea841026ef3842c74c77f736d2d08838"
server_key_iv: "b9ad8c08f9f31d8290560c656f89aa8f"
server_key_version: "encryptionkey@0"
I20260619 06:52:36.998600 3722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "333e548bbc634330a7f84ca932202180"
format_stamp: "Formatted at 2026-06-19 06:52:36 on dist-test-slave-9x3b"
server_key: "ea841026ef3842c74c77f736d2d08838"
server_key_iv: "b9ad8c08f9f31d8290560c656f89aa8f"
server_key_version: "encryptionkey@0"
I20260619 06:52:37.002208 3722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:52:37.004608 3737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:37.005796 3722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:37.005941 3722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "333e548bbc634330a7f84ca932202180"
format_stamp: "Formatted at 2026-06-19 06:52:36 on dist-test-slave-9x3b"
server_key: "ea841026ef3842c74c77f736d2d08838"
server_key_iv: "b9ad8c08f9f31d8290560c656f89aa8f"
server_key_version: "encryptionkey@0"
I20260619 06:52:37.006063 3722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:37.021343 3722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:37.024444 3722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:37.024645 3722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:37.031762 3722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42425
I20260619 06:52:37.031805 3789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42425 every 8 connection(s)
I20260619 06:52:37.032707 3722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:37.035351 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:37.037547 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 3722
I20260619 06:52:37.037672 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:37.037974 14803 external_mini_cluster.cc:1428] Setting key c0ae3a0cc51268ed665ddd1cf8faa212
I20260619 06:52:37.041462 3790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180: Bootstrap starting.
I20260619 06:52:37.043541 3790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:37.044271 3790 log.cc:824] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:37.046440 3790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180: No bootstrap required, opened a new log
I20260619 06:52:37.047096 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:37.039429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:57961 (local address 127.0.0.1:42425)
0619 06:52:37.039854 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:37.039869 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:37.039903 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:52:37.040647 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:37.040665 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:37.041001 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:37.041340 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:37.041731 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:37.042671 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:37.043588 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:37.044108 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:37.044284 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:37.045155 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:37.045163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:37.045178 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:37.045680 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:37.045747 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:37.045777 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:37.045911 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:37.046046 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 06:52:37.046170 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":155,"threads_started":1}
I20260619 06:52:37.049917 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:37.050477 3790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333e548bbc634330a7f84ca932202180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } }
I20260619 06:52:37.050676 3790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:37.050735 3790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333e548bbc634330a7f84ca932202180, State: Initialized, Role: FOLLOWER
I20260619 06:52:37.051156 3790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333e548bbc634330a7f84ca932202180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } }
I20260619 06:52:37.051298 3790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:37.051386 3790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:37.051496 3790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:37.052603 3790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333e548bbc634330a7f84ca932202180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } }
I20260619 06:52:37.052968 3790 leader_election.cc:302] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 333e548bbc634330a7f84ca932202180; no voters:
I20260619 06:52:37.053290 3790 leader_election.cc:288] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:37.053417 3800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:37.053640 3800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [term 1 LEADER]: Becoming Leader. State: Replica: 333e548bbc634330a7f84ca932202180, State: Running, Role: LEADER
I20260619 06:52:37.053968 3800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333e548bbc634330a7f84ca932202180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } }
I20260619 06:52:37.054524 3790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:37.055043 3800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333e548bbc634330a7f84ca932202180. Latest consensus state: current_term: 1 leader_uuid: "333e548bbc634330a7f84ca932202180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333e548bbc634330a7f84ca932202180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } } }
I20260619 06:52:37.055161 3800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:37.055596 3806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:37.055596 3801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "333e548bbc634330a7f84ca932202180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333e548bbc634330a7f84ca932202180" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42425 } } }
I20260619 06:52:37.055819 3801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:37.057142 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:37.052063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:39915 (local address 127.0.0.1:42425)
0619 06:52:37.052181 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:37.052185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:37.052339 (+ 154us) server_negotiation.cc:407] Connection header received
0619 06:52:37.052430 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:37.052434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:37.052490 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:37.052573 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:37.052828 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:37.053566 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:37.054674 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:37.055008 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:37.056093 (+ 1085us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:37.056253 (+ 160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:37.056258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:37.056260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:37.056274 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:37.056313 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:37.056327 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:37.056889 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:37.056952 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 06:52:37.056990 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:52:37.059341 3806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:37.066067 3806 catalog_manager.cc:1347] Generated new cluster ID: c95eda2ee2f940eeb78017c3b59ecd9e
I20260619 06:52:37.066150 3806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:37.078689 3806 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:37.079881 3806 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:37.093128 3806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 333e548bbc634330a7f84ca932202180: Generated new TSK 0
I20260619 06:52:37.093923 3806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:37.106106 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 3722
W20260619 06:52:37.111269 3707 connection.cc:535] client connection to 127.0.0.1:42425 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42425): Connection reset by peer (error 104)
2026-06-19T06:52:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:37 dist-test-slave-9x3b krb5kdc[3821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:37 dist-test-slave-9x3b krb5kdc[3821](info): set up 2 sockets
Jun 19 06:52:37 dist-test-slave-9x3b krb5kdc[3821](info): commencing operation
krb5kdc: starting...
W20260619 06:52:38.135831 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:38 dist-test-slave-9x3b krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:52:38.250664 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:38.356149 3834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:38.356402 3834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:38.356451 3834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:38.360268 3834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:38.360316 3834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:38.360335 3834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:38.360352 3834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:38.360368 3834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:38.365375 3834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:38.366283 3834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:38.367203 3834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:38.375253 3842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:38.375247 3839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:38.375247 3840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:38.375523 3834 server_base.cc:1034] running on GCE node
I20260619 06:52:38.376160 3834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:38.377170 3834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:38.378326 3834 hybrid_clock.cc:648] HybridClock initialized: now 1781851958378299 us; error 44 us; skew 500 ppm
Jun 19 06:52:38 dist-test-slave-9x3b krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:38.385883 3834 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:52:38.386818 3834 webserver.cc:460] Webserver started at http://127.0.0.1:40051/ using document root <none> and password file <none>
I20260619 06:52:38.387367 3834 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:38.387426 3834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:38.387591 3834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:38.389674 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "a43d5594870e4e1dac68138adabf4f98"
format_stamp: "Formatted at 2026-06-19 06:52:38 on dist-test-slave-9x3b"
server_key: "b6425c899aaa224642be1ebb506f63a3"
server_key_iv: "dfd7560a2410b0b6a531fd474e5501b0"
server_key_version: "encryptionkey@0"
I20260619 06:52:38.390194 3834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "a43d5594870e4e1dac68138adabf4f98"
format_stamp: "Formatted at 2026-06-19 06:52:38 on dist-test-slave-9x3b"
server_key: "b6425c899aaa224642be1ebb506f63a3"
server_key_iv: "dfd7560a2410b0b6a531fd474e5501b0"
server_key_version: "encryptionkey@0"
I20260619 06:52:38.393848 3834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:38.396246 3849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:38.397444 3834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 06:52:38.397575 3834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "a43d5594870e4e1dac68138adabf4f98"
format_stamp: "Formatted at 2026-06-19 06:52:38 on dist-test-slave-9x3b"
server_key: "b6425c899aaa224642be1ebb506f63a3"
server_key_iv: "dfd7560a2410b0b6a531fd474e5501b0"
server_key_version: "encryptionkey@0"
I20260619 06:52:38.397680 3834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:38.402264 3834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:38.405087 3834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:38.405283 3834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:38.411966 3834 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33753
I20260619 06:52:38.411989 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33753 every 8 connection(s)
I20260619 06:52:38.412886 3834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:38.415594 3902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:38.416138 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 3834
I20260619 06:52:38.416291 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:38.416594 14803 external_mini_cluster.cc:1428] Setting key 9c6876a3b080086c689434917a454989
I20260619 06:52:38.421967 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:38.417932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:52763 (local address 127.0.0.1:33753)
0619 06:52:38.418335 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:38.418353 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:38.418397 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:52:38.419207 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:38.419230 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:38.419575 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:38.419947 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:38.420631 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:38.420636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:38.420651 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:38.420663 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:38.420722 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:38.420746 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:38.420883 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:38.420960 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 06:52:38.421113 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260619 06:52:38.422235 3902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98: Bootstrap starting.
I20260619 06:52:38.424644 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:38.424667 3902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:38.425357 3902 log.cc:824] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:38.427695 3902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98: No bootstrap required, opened a new log
I20260619 06:52:38.428532 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:38.426756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:39373 (local address 127.0.0.1:33753)
0619 06:52:38.426883 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:38.426887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:38.427066 (+ 179us) server_negotiation.cc:407] Connection header received
0619 06:52:38.427190 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:38.427192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:38.427217 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:38.427285 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:38.428096 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:38.428099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:38.428102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:38.428113 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:38.428139 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:38.428150 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:38.428243 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:38.428378 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 06:52:38.428418 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:52:38.431335 3902 raft_consensus.cc:348] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43d5594870e4e1dac68138adabf4f98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33753 } }
I20260619 06:52:38.431547 3902 raft_consensus.cc:374] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:38.431607 3902 raft_consensus.cc:729] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43d5594870e4e1dac68138adabf4f98, State: Initialized, Role: FOLLOWER
I20260619 06:52:38.432076 3902 consensus_queue.cc:260] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43d5594870e4e1dac68138adabf4f98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33753 } }
I20260619 06:52:38.432196 3902 raft_consensus.cc:388] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:38.432279 3902 raft_consensus.cc:482] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:38.432376 3902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:38.433447 3902 raft_consensus.cc:504] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43d5594870e4e1dac68138adabf4f98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33753 } }
I20260619 06:52:38.433773 3902 leader_election.cc:302] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a43d5594870e4e1dac68138adabf4f98; no voters:
I20260619 06:52:38.434090 3902 leader_election.cc:288] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:38.434213 3912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:38.434485 3912 raft_consensus.cc:686] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [term 1 LEADER]: Becoming Leader. State: Replica: a43d5594870e4e1dac68138adabf4f98, State: Running, Role: LEADER
I20260619 06:52:38.434803 3912 consensus_queue.cc:237] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43d5594870e4e1dac68138adabf4f98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33753 } }
I20260619 06:52:38.435256 3902 sys_catalog.cc:564] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:38.436434 3913 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a43d5594870e4e1dac68138adabf4f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43d5594870e4e1dac68138adabf4f98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33753 } } }
I20260619 06:52:38.436555 3913 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:38.436861 3914 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a43d5594870e4e1dac68138adabf4f98. Latest consensus state: current_term: 1 leader_uuid: "a43d5594870e4e1dac68138adabf4f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43d5594870e4e1dac68138adabf4f98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33753 } } }
I20260619 06:52:38.436947 3914 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:38.437053 3921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:38.439926 3921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:38.444979 3921 catalog_manager.cc:1347] Generated new cluster ID: bfa0ed36539e49948c21e5b6c2a255d8
I20260619 06:52:38.445052 3921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:38.464102 3921 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:38.464416 3921 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:38.474617 3921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a43d5594870e4e1dac68138adabf4f98: Generated new TSK 0
I20260619 06:52:38.475281 3921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:38.484906 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 3834
2026-06-19T06:52:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:38 dist-test-slave-9x3b krb5kdc[3933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:38 dist-test-slave-9x3b krb5kdc[3933](info): set up 2 sockets
Jun 19 06:52:38 dist-test-slave-9x3b krb5kdc[3933](info): commencing operation
krb5kdc: starting...
W20260619 06:52:39.511807 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:39 dist-test-slave-9x3b krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:39.626528 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:42763
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:42763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:39.732652 3946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:39.732923 3946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:39.733011 3946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:39.737047 3946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:39.737105 3946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:39.737125 3946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:39.737141 3946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:39.737157 3946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:39.742343 3946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:42763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:42763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:39.743366 3946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:39.744449 3946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:39.753100 3951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:39.753304 3954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:39.753208 3946 server_base.cc:1034] running on GCE node
W20260619 06:52:39.753100 3952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:39.753901 3946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:39.754964 3946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:39.756147 3946 hybrid_clock.cc:648] HybridClock initialized: now 1781851959756119 us; error 42 us; skew 500 ppm
Jun 19 06:52:39 dist-test-slave-9x3b krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851959, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:39.758661 3946 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:39.759471 3946 webserver.cc:460] Webserver started at http://127.14.116.254:33655/ using document root <none> and password file <none>
I20260619 06:52:39.760054 3946 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:39.760143 3946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:39.760365 3946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:39.762315 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "0338b24640ca4c32889c3f63beb824f0"
format_stamp: "Formatted at 2026-06-19 06:52:39 on dist-test-slave-9x3b"
server_key: "0bf4216c96267d7e14be3d471c978d95"
server_key_iv: "c24964b961143563838983a6376caf72"
server_key_version: "encryptionkey@0"
I20260619 06:52:39.762868 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "0338b24640ca4c32889c3f63beb824f0"
format_stamp: "Formatted at 2026-06-19 06:52:39 on dist-test-slave-9x3b"
server_key: "0bf4216c96267d7e14be3d471c978d95"
server_key_iv: "c24964b961143563838983a6376caf72"
server_key_version: "encryptionkey@0"
I20260619 06:52:39.766353 3946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:52:39.768702 3961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:39.769888 3946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:52:39.770022 3946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "0338b24640ca4c32889c3f63beb824f0"
format_stamp: "Formatted at 2026-06-19 06:52:39 on dist-test-slave-9x3b"
server_key: "0bf4216c96267d7e14be3d471c978d95"
server_key_iv: "c24964b961143563838983a6376caf72"
server_key_version: "encryptionkey@0"
I20260619 06:52:39.770131 3946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:39.784269 3946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:39.786928 3946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:39.787123 3946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:39.793995 3946 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:42763
I20260619 06:52:39.794018 4013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:42763 every 8 connection(s)
I20260619 06:52:39.794906 3946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:39.797487 4014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:39.802151 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 3946
I20260619 06:52:39.802302 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:39.802589 14803 external_mini_cluster.cc:1428] Setting key 21de0b46bc0c57543e94176d36bda7bf
I20260619 06:52:39.802843 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0: Bootstrap starting.
I20260619 06:52:39.805311 4014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:39.806085 4014 log.cc:824] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:39.807964 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0: No bootstrap required, opened a new log
Jun 19 06:52:39 dist-test-slave-9x3b krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:39.811942 4014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b24640ca4c32889c3f63beb824f0" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42763 } }
I20260619 06:52:39.812181 4014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:39.812243 4014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0338b24640ca4c32889c3f63beb824f0, State: Initialized, Role: FOLLOWER
I20260619 06:52:39.812747 4014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b24640ca4c32889c3f63beb824f0" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42763 } }
I20260619 06:52:39.812873 4014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:39.812975 4014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:39.813082 4014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:39.814018 4014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b24640ca4c32889c3f63beb824f0" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42763 } }
I20260619 06:52:39.814401 4014 leader_election.cc:302] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0338b24640ca4c32889c3f63beb824f0; no voters:
I20260619 06:52:39.814762 4014 leader_election.cc:288] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:39.814817 4019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:39.815076 4019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [term 1 LEADER]: Becoming Leader. State: Replica: 0338b24640ca4c32889c3f63beb824f0, State: Running, Role: LEADER
I20260619 06:52:39.815407 4019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b24640ca4c32889c3f63beb824f0" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42763 } }
I20260619 06:52:39.816129 4014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:39.817011 4020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0338b24640ca4c32889c3f63beb824f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b24640ca4c32889c3f63beb824f0" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42763 } } }
I20260619 06:52:39.817123 4020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:39.817476 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0338b24640ca4c32889c3f63beb824f0. Latest consensus state: current_term: 1 leader_uuid: "0338b24640ca4c32889c3f63beb824f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0338b24640ca4c32889c3f63beb824f0" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42763 } } }
I20260619 06:52:39.817533 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:39.804023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46835 (local address 127.14.116.254:42763)
0619 06:52:39.804436 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:39.804452 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:39.804485 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:52:39.805099 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:39.805115 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:39.805490 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:39.805813 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:39.806599 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:39.807563 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:39.808542 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:39.809024 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:39.809200 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:39.812034 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:39.812039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:39.812051 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:39.812077 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:39.813871 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:39.814442 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:39.814447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:39.814451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:39.814625 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:39.815024 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:39.815027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:39.815029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:39.815507 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:39.815713 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:39.816239 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 06:52:39.816487 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":150,"threads_started":1}
I20260619 06:52:39.818631 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:39.820663 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 06:52:39.820917 4034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:39.821002 4034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:39.821058 4028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:39.821857 4028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:39.827489 4028 catalog_manager.cc:1347] Generated new cluster ID: bd46b4b0c6964903af7ffd2745ec60bb
I20260619 06:52:39.827548 4028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:39.830821 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:39.822597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50045 (local address 127.14.116.254:42763)
0619 06:52:39.822721 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:39.822725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:39.823016 (+ 291us) server_negotiation.cc:407] Connection header received
0619 06:52:39.823158 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:39.823162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:39.823209 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:39.823295 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:39.824002 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:39.824514 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:39.825286 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:39.825539 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:39.825654 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:39.827408 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:39.827412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:39.827415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:39.827438 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:39.829082 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:39.829551 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:39.829554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:39.829556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:39.829634 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:39.829961 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:39.829965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:39.829966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:39.830132 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:39.830281 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:39.830551 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:52:39.830648 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:52:39.837467 4028 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:39.838315 4028 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:39.841899 4028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0338b24640ca4c32889c3f63beb824f0: Generated new TSK 0
I20260619 06:52:39.842666 4028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:39.852463 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 3946
W20260619 06:52:39.858777 3931 connection.cc:535] client connection to 127.14.116.254:42763 recv error: Network error: failed to read from TLS socket (remote: 127.14.116.254:42763): Connection reset by peer (error 104)
2026-06-19T06:52:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:39 dist-test-slave-9x3b krb5kdc[4045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:39 dist-test-slave-9x3b krb5kdc[4045](info): set up 2 sockets
Jun 19 06:52:39 dist-test-slave-9x3b krb5kdc[4045](info): commencing operation
krb5kdc: starting...
W20260619 06:52:40.887959 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:40 dist-test-slave-9x3b krb5kdc[4045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:41.012889 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:45913
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:45913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:41.126463 4058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:41.126730 4058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:41.126801 4058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:41.130954 4058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:41.131024 4058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:41.131044 4058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:41.131062 4058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:41.131078 4058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:41.136561 4058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:45913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:45913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:41.137609 4058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:41.138662 4058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:41.147171 4063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:41.147172 4066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:41.147172 4064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:41.147466 4058 server_base.cc:1034] running on GCE node
I20260619 06:52:41.148102 4058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:41.149233 4058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:41.150440 4058 hybrid_clock.cc:648] HybridClock initialized: now 1781851961150423 us; error 37 us; skew 500 ppm
Jun 19 06:52:41 dist-test-slave-9x3b krb5kdc[4045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851961, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:41.153425 4058 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:41.154371 4058 webserver.cc:460] Webserver started at http://127.14.116.254:32807/ using document root <none> and password file <none>
I20260619 06:52:41.154935 4058 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:41.155026 4058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:41.155256 4058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:41.157349 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "b48ea7267c634ab5bac79438467d1108"
format_stamp: "Formatted at 2026-06-19 06:52:41 on dist-test-slave-9x3b"
server_key: "aabc59760a4dfb2d0388aa2db5c380c2"
server_key_iv: "b55ac362a10a1691cd12b02b5e6a2d17"
server_key_version: "encryptionkey@0"
I20260619 06:52:41.157914 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "b48ea7267c634ab5bac79438467d1108"
format_stamp: "Formatted at 2026-06-19 06:52:41 on dist-test-slave-9x3b"
server_key: "aabc59760a4dfb2d0388aa2db5c380c2"
server_key_iv: "b55ac362a10a1691cd12b02b5e6a2d17"
server_key_version: "encryptionkey@0"
I20260619 06:52:41.161720 4058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 06:52:41.164211 4073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:41.165488 4058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:41.165633 4058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "b48ea7267c634ab5bac79438467d1108"
format_stamp: "Formatted at 2026-06-19 06:52:41 on dist-test-slave-9x3b"
server_key: "aabc59760a4dfb2d0388aa2db5c380c2"
server_key_iv: "b55ac362a10a1691cd12b02b5e6a2d17"
server_key_version: "encryptionkey@0"
I20260619 06:52:41.165751 4058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:41.206949 4058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:41.210412 4058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:41.210621 4058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:41.217726 4058 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:45913
I20260619 06:52:41.217754 4125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:45913 every 8 connection(s)
I20260619 06:52:41.218662 4058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:41.219489 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 4058
I20260619 06:52:41.219672 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:41.219969 14803 external_mini_cluster.cc:1428] Setting key 8096735c2067d10729a280079fe9aae8
I20260619 06:52:41.221685 4126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:41.228355 4126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108: Bootstrap starting.
I20260619 06:52:41.229918 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:41.221360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38195 (local address 127.14.116.254:45913)
0619 06:52:41.221783 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:41.221802 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:41.221842 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:52:41.222659 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:41.222680 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:41.223091 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:41.223454 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:41.223871 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:41.225330 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:41.226237 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:41.226917 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:41.227070 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:41.227982 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:41.227988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:41.228003 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:41.228490 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:41.228554 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:41.228582 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:41.228711 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:41.228882 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 06:52:41.229006 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260619 06:52:41.230973 4126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:41.231786 4126 log.cc:824] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:41.232848 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:41.234337 4126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108: No bootstrap required, opened a new log
I20260619 06:52:41.239046 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:41.235040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37339 (local address 127.14.116.254:45913)
0619 06:52:41.235158 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:41.235162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:41.235314 (+ 152us) server_negotiation.cc:407] Connection header received
0619 06:52:41.235390 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:41.235393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:41.235433 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:41.235516 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:41.235811 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:41.236633 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:41.237402 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:41.237732 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:41.237840 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:41.238607 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:41.238611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:41.238613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:41.238626 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:41.238651 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:41.238663 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:41.238751 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:41.238841 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 06:52:41.238870 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:52:41.238919 4126 raft_consensus.cc:348] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48ea7267c634ab5bac79438467d1108" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 45913 } }
I20260619 06:52:41.239202 4126 raft_consensus.cc:374] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:41.239284 4126 raft_consensus.cc:729] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48ea7267c634ab5bac79438467d1108, State: Initialized, Role: FOLLOWER
I20260619 06:52:41.239773 4126 consensus_queue.cc:260] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48ea7267c634ab5bac79438467d1108" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 45913 } }
I20260619 06:52:41.239940 4126 raft_consensus.cc:388] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:41.240005 4126 raft_consensus.cc:482] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:41.240098 4126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:41.240985 4126 raft_consensus.cc:504] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48ea7267c634ab5bac79438467d1108" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 45913 } }
I20260619 06:52:41.241318 4126 leader_election.cc:302] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b48ea7267c634ab5bac79438467d1108; no voters:
I20260619 06:52:41.241631 4126 leader_election.cc:288] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:41.241732 4136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:41.241977 4136 raft_consensus.cc:686] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [term 1 LEADER]: Becoming Leader. State: Replica: b48ea7267c634ab5bac79438467d1108, State: Running, Role: LEADER
I20260619 06:52:41.242316 4136 consensus_queue.cc:237] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48ea7267c634ab5bac79438467d1108" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 45913 } }
I20260619 06:52:41.242843 4126 sys_catalog.cc:564] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:41.243937 4138 sys_catalog.cc:455] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b48ea7267c634ab5bac79438467d1108. Latest consensus state: current_term: 1 leader_uuid: "b48ea7267c634ab5bac79438467d1108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48ea7267c634ab5bac79438467d1108" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 45913 } } }
I20260619 06:52:41.244078 4138 sys_catalog.cc:458] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:41.244423 4145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:41.244942 4137 sys_catalog.cc:455] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b48ea7267c634ab5bac79438467d1108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b48ea7267c634ab5bac79438467d1108" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 45913 } } }
I20260619 06:52:41.245138 4137 sys_catalog.cc:458] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:41.247118 4145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:41.252383 4145 catalog_manager.cc:1347] Generated new cluster ID: 3551ae5b82c94aedb046f945e655910f
I20260619 06:52:41.252466 4145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:41.263967 4145 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:41.265034 4145 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:41.270730 4145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b48ea7267c634ab5bac79438467d1108: Generated new TSK 0
I20260619 06:52:41.271394 4145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:41.275010 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 4058
W20260619 06:52:41.280201 4043 connection.cc:535] client connection to 127.14.116.254:45913 recv error: Network error: failed to read from TLS socket (remote: 127.14.116.254:45913): Connection reset by peer (error 104)
2026-06-19T06:52:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:41 dist-test-slave-9x3b krb5kdc[4157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:41 dist-test-slave-9x3b krb5kdc[4157](info): set up 2 sockets
Jun 19 06:52:41 dist-test-slave-9x3b krb5kdc[4157](info): commencing operation
krb5kdc: starting...
W20260619 06:52:42.303818 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:42 dist-test-slave-9x3b krb5kdc[4157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:42.420262 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:37479
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:37479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:42.529826 4170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:42.530112 4170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:42.530155 4170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:42.534274 4170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:42.534339 4170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:42.534390 4170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:42.534428 4170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:42.534466 4170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:42.539987 4170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:37479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:37479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:42.541024 4170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:42.542021 4170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:42.550460 4178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:42.550455 4175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:42.550673 4170 server_base.cc:1034] running on GCE node
W20260619 06:52:42.550455 4176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:42.551347 4170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:42.552515 4170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:42.553702 4170 hybrid_clock.cc:648] HybridClock initialized: now 1781851962553672 us; error 43 us; skew 500 ppm
Jun 19 06:52:42 dist-test-slave-9x3b krb5kdc[4157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851962, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:42.556541 4170 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:42.557471 4170 webserver.cc:460] Webserver started at http://127.14.116.254:36707/ using document root <none> and password file <none>
I20260619 06:52:42.558044 4170 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:42.558135 4170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:42.558373 4170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:42.560470 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "e5172b32a4984dbaae5d691a7c766757"
format_stamp: "Formatted at 2026-06-19 06:52:42 on dist-test-slave-9x3b"
server_key: "502c5e33663a2625b0325b47c8f4ba9c"
server_key_iv: "f9cecd335152422b93b1dce1f71a52c9"
server_key_version: "encryptionkey@0"
I20260619 06:52:42.561149 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "e5172b32a4984dbaae5d691a7c766757"
format_stamp: "Formatted at 2026-06-19 06:52:42 on dist-test-slave-9x3b"
server_key: "502c5e33663a2625b0325b47c8f4ba9c"
server_key_iv: "f9cecd335152422b93b1dce1f71a52c9"
server_key_version: "encryptionkey@0"
I20260619 06:52:42.564975 4170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:52:42.567369 4185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:42.568555 4170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:42.568696 4170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "e5172b32a4984dbaae5d691a7c766757"
format_stamp: "Formatted at 2026-06-19 06:52:42 on dist-test-slave-9x3b"
server_key: "502c5e33663a2625b0325b47c8f4ba9c"
server_key_iv: "f9cecd335152422b93b1dce1f71a52c9"
server_key_version: "encryptionkey@0"
I20260619 06:52:42.568806 4170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:42.573278 4170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:42.576624 4170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:42.576817 4170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:42.583489 4170 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:37479
I20260619 06:52:42.583546 4237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:37479 every 8 connection(s)
I20260619 06:52:42.584452 4170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:42.586279 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 4170
I20260619 06:52:42.586427 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:42.586699 14803 external_mini_cluster.cc:1428] Setting key 7a0674194c100c0f9a18716de2de90b6
I20260619 06:52:42.587378 4238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:42.592257 4241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:42.588087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57835 (local address 127.14.116.254:37479)
0619 06:52:42.588508 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:42.588526 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:42.588569 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:52:42.589440 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:42.589463 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:42.589826 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:42.590198 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:42.590847 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:42.590852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:42.590867 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:42.590879 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:42.590937 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:42.590964 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:42.591112 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:42.591184 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 06:52:42.591329 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260619 06:52:42.594098 4238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757: Bootstrap starting.
I20260619 06:52:42.595063 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:42.596470 4238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:42.597196 4238 log.cc:824] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:42.598948 4241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:42.597234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33733 (local address 127.14.116.254:37479)
0619 06:52:42.597360 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:42.597364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:42.597581 (+ 217us) server_negotiation.cc:407] Connection header received
0619 06:52:42.597726 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:42.597729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:42.597766 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:42.597850 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:42.598578 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:42.598581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:42.598584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:42.598594 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:42.598620 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:42.598631 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:42.598725 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:42.598798 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 06:52:42.598836 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:52:42.599046 4238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757: No bootstrap required, opened a new log
I20260619 06:52:42.602593 4238 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5172b32a4984dbaae5d691a7c766757" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37479 } }
I20260619 06:52:42.602821 4238 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:42.602857 4238 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5172b32a4984dbaae5d691a7c766757, State: Initialized, Role: FOLLOWER
I20260619 06:52:42.603216 4238 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5172b32a4984dbaae5d691a7c766757" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37479 } }
I20260619 06:52:42.603330 4238 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:42.603365 4238 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:42.603431 4238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:42.604295 4238 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5172b32a4984dbaae5d691a7c766757" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37479 } }
I20260619 06:52:42.604585 4238 leader_election.cc:302] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5172b32a4984dbaae5d691a7c766757; no voters:
I20260619 06:52:42.604848 4238 leader_election.cc:288] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:42.604929 4248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:42.605168 4248 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [term 1 LEADER]: Becoming Leader. State: Replica: e5172b32a4984dbaae5d691a7c766757, State: Running, Role: LEADER
I20260619 06:52:42.605502 4248 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5172b32a4984dbaae5d691a7c766757" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37479 } }
I20260619 06:52:42.606063 4238 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:42.607254 4250 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5172b32a4984dbaae5d691a7c766757. Latest consensus state: current_term: 1 leader_uuid: "e5172b32a4984dbaae5d691a7c766757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5172b32a4984dbaae5d691a7c766757" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37479 } } }
I20260619 06:52:42.607254 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5172b32a4984dbaae5d691a7c766757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5172b32a4984dbaae5d691a7c766757" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 37479 } } }
I20260619 06:52:42.607412 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:42.607412 4250 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:42.607867 4257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:42.611122 4257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:42.616737 4257 catalog_manager.cc:1347] Generated new cluster ID: 50c1ff2808f14c86bf8a9a0950c90de7
I20260619 06:52:42.616858 4257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:42.624992 4257 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:42.625308 4257 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:42.632665 4257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5172b32a4984dbaae5d691a7c766757: Generated new TSK 0
I20260619 06:52:42.633307 4257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:42.640844 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 4170
2026-06-19T06:52:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:42 dist-test-slave-9x3b krb5kdc[4269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:42 dist-test-slave-9x3b krb5kdc[4269](info): set up 2 sockets
Jun 19 06:52:42 dist-test-slave-9x3b krb5kdc[4269](info): commencing operation
krb5kdc: starting...
W20260619 06:52:43.675953 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:43 dist-test-slave-9x3b krb5kdc[4269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:43.792268 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:41933
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:46437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:41933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:43.897521 4282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:43.897791 4282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:43.897874 4282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:43.901861 4282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:43.901927 4282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:43.901947 4282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:43.902016 4282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:43.902046 4282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:43.907272 4282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:41933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:41933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:43.908353 4282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:43.909371 4282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:43.917646 4288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:43.917637 4287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:43.917711 4282 server_base.cc:1034] running on GCE node
W20260619 06:52:43.917637 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:43.918367 4282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:43.919462 4282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:43.920632 4282 hybrid_clock.cc:648] HybridClock initialized: now 1781851963920622 us; error 28 us; skew 500 ppm
Jun 19 06:52:43 dist-test-slave-9x3b krb5kdc[4269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851963, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:43.927984 4282 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:43.928857 4282 webserver.cc:460] Webserver started at http://127.14.116.254:43423/ using document root <none> and password file <none>
I20260619 06:52:43.929371 4282 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:43.929426 4282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:43.929596 4282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:43.931551 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "c47a78505e964583b2438dac9e599b4b"
format_stamp: "Formatted at 2026-06-19 06:52:43 on dist-test-slave-9x3b"
server_key: "3003b575db8580999506253261480b33"
server_key_iv: "79c23a67487ae3500e7b84a2a0d1b055"
server_key_version: "encryptionkey@0"
I20260619 06:52:43.932132 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "c47a78505e964583b2438dac9e599b4b"
format_stamp: "Formatted at 2026-06-19 06:52:43 on dist-test-slave-9x3b"
server_key: "3003b575db8580999506253261480b33"
server_key_iv: "79c23a67487ae3500e7b84a2a0d1b055"
server_key_version: "encryptionkey@0"
I20260619 06:52:43.935716 4282 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:52:43.938007 4297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:43.939273 4282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:43.939414 4282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "c47a78505e964583b2438dac9e599b4b"
format_stamp: "Formatted at 2026-06-19 06:52:43 on dist-test-slave-9x3b"
server_key: "3003b575db8580999506253261480b33"
server_key_iv: "79c23a67487ae3500e7b84a2a0d1b055"
server_key_version: "encryptionkey@0"
I20260619 06:52:43.939551 4282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:43.951010 4282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:43.953698 4282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:43.953891 4282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:43.960740 4282 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:41933
I20260619 06:52:43.960812 4349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:41933 every 8 connection(s)
I20260619 06:52:43.961668 4282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:43.964413 4350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:43.968009 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 4282
I20260619 06:52:43.968144 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:43.968411 14803 external_mini_cluster.cc:1428] Setting key 1a299f5ff1afaab3bf2c0f184b622119
I20260619 06:52:43.970412 4350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b: Bootstrap starting.
I20260619 06:52:43.973080 4350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:43.973903 4350 log.cc:824] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:43.976094 4350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b: No bootstrap required, opened a new log
Jun 19 06:52:43 dist-test-slave-9x3b krb5kdc[4269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:43.980321 4350 raft_consensus.cc:348] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a78505e964583b2438dac9e599b4b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41933 } }
I20260619 06:52:43.980641 4350 raft_consensus.cc:374] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:43.980703 4350 raft_consensus.cc:729] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47a78505e964583b2438dac9e599b4b, State: Initialized, Role: FOLLOWER
I20260619 06:52:43.981227 4350 consensus_queue.cc:260] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a78505e964583b2438dac9e599b4b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41933 } }
I20260619 06:52:43.981397 4350 raft_consensus.cc:388] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:43.981469 4350 raft_consensus.cc:482] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:43.981565 4350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:43.982735 4350 raft_consensus.cc:504] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a78505e964583b2438dac9e599b4b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41933 } }
I20260619 06:52:43.983093 4350 leader_election.cc:302] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c47a78505e964583b2438dac9e599b4b; no voters:
I20260619 06:52:43.983403 4350 leader_election.cc:288] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:43.983516 4355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:43.983687 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:43.969774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59487 (local address 127.14.116.254:41933)
0619 06:52:43.970181 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:43.970200 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:43.970241 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:52:43.971040 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:43.971064 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:43.971468 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:43.971876 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:43.972496 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:43.974007 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:43.975055 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:43.975753 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:43.975914 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:43.978321 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:43.978336 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:43.978359 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:43.978398 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:43.980398 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:43.980935 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:43.980944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:43.980949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:43.981082 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:43.981349 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:43.981353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:43.981354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:43.981903 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:43.982083 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:43.982356 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:52:43.982682 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":105,"threads_started":1}
I20260619 06:52:43.983793 4355 raft_consensus.cc:686] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [term 1 LEADER]: Becoming Leader. State: Replica: c47a78505e964583b2438dac9e599b4b, State: Running, Role: LEADER
I20260619 06:52:43.984131 4355 consensus_queue.cc:237] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a78505e964583b2438dac9e599b4b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41933 } }
I20260619 06:52:43.984652 4350 sys_catalog.cc:564] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:43.985316 4355 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c47a78505e964583b2438dac9e599b4b. Latest consensus state: current_term: 1 leader_uuid: "c47a78505e964583b2438dac9e599b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a78505e964583b2438dac9e599b4b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41933 } } }
I20260619 06:52:43.985307 4356 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c47a78505e964583b2438dac9e599b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47a78505e964583b2438dac9e599b4b" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 41933 } } }
I20260619 06:52:43.985468 4355 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:43.985478 4356 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:43.986845 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 06:52:43.990135 4370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:43.990222 4370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:43.990316 4362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:43.991019 4362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:43.996863 4362 catalog_manager.cc:1347] Generated new cluster ID: d4075859a2934da0aa3f6a45c4c1a3c1
I20260619 06:52:43.996960 4362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:43.997519 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:43.989046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32987 (local address 127.14.116.254:41933)
0619 06:52:43.989165 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:43.989169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:43.989400 (+ 231us) server_negotiation.cc:407] Connection header received
0619 06:52:43.989518 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:43.989521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:43.989565 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:43.989648 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:43.990504 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:43.991028 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:43.992163 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:43.992496 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:43.992604 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:43.993974 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:43.993978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:43.993981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:43.994012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:43.995724 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:43.996135 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:43.996140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:43.996142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:43.996223 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:43.996516 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:43.996520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:43.996522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:43.996737 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:43.996884 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:43.997242 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 06:52:43.997370 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:52:44.010728 4362 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:44.011706 4362 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:44.017664 4362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c47a78505e964583b2438dac9e599b4b: Generated new TSK 0
I20260619 06:52:44.018314 4362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:44.024708 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 4282
2026-06-19T06:52:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:44 dist-test-slave-9x3b krb5kdc[4381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:44 dist-test-slave-9x3b krb5kdc[4381](info): set up 2 sockets
Jun 19 06:52:44 dist-test-slave-9x3b krb5kdc[4381](info): commencing operation
krb5kdc: starting...
W20260619 06:52:45.068559 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:45 dist-test-slave-9x3b krb5kdc[4381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:45.184594 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:33821
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:45.291100 4394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:45.291347 4394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:45.291393 4394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:45.295341 4394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:45.295403 4394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:45.295421 4394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:45.295472 4394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:45.295507 4394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:45.300714 4394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:33821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:33821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:45.301702 4394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:45.302673 4394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:45.311014 4399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:45.310945 4402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:45.310936 4400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:45.311211 4394 server_base.cc:1034] running on GCE node
I20260619 06:52:45.311920 4394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:45.312963 4394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:45.314131 4394 hybrid_clock.cc:648] HybridClock initialized: now 1781851965314113 us; error 30 us; skew 500 ppm
Jun 19 06:52:45 dist-test-slave-9x3b krb5kdc[4381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851965, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:45.316753 4394 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:45.317580 4394 webserver.cc:460] Webserver started at http://127.14.116.254:39741/ using document root <none> and password file <none>
I20260619 06:52:45.318064 4394 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:45.318120 4394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:45.318272 4394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:45.320287 4394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "75c61a74e85f4af1ae369beab260fb7c"
format_stamp: "Formatted at 2026-06-19 06:52:45 on dist-test-slave-9x3b"
server_key: "633b425cecbe7234d3b246ec88b59c76"
server_key_iv: "f203fe6728b82d575ba4e1b833a4743e"
server_key_version: "encryptionkey@0"
I20260619 06:52:45.320765 4394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "75c61a74e85f4af1ae369beab260fb7c"
format_stamp: "Formatted at 2026-06-19 06:52:45 on dist-test-slave-9x3b"
server_key: "633b425cecbe7234d3b246ec88b59c76"
server_key_iv: "f203fe6728b82d575ba4e1b833a4743e"
server_key_version: "encryptionkey@0"
I20260619 06:52:45.324167 4394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:52:45.326328 4409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:45.327466 4394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:45.327641 4394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "75c61a74e85f4af1ae369beab260fb7c"
format_stamp: "Formatted at 2026-06-19 06:52:45 on dist-test-slave-9x3b"
server_key: "633b425cecbe7234d3b246ec88b59c76"
server_key_iv: "f203fe6728b82d575ba4e1b833a4743e"
server_key_version: "encryptionkey@0"
I20260619 06:52:45.327756 4394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:45.365486 4394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:45.368290 4394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:45.368512 4394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:45.375599 4394 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:33821
I20260619 06:52:45.375643 4461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:33821 every 8 connection(s)
I20260619 06:52:45.376547 4394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:45.379112 4462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:45.380537 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 4394
I20260619 06:52:45.380676 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:45.380944 14803 external_mini_cluster.cc:1428] Setting key 49116876c694581ef9986cc6a29fb65c
I20260619 06:52:45.385689 4462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c: Bootstrap starting.
I20260619 06:52:45.388254 4462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:45.389122 4462 log.cc:824] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:45.390743 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:45.382272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41947 (local address 127.14.116.254:33821)
0619 06:52:45.382674 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:45.382696 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:45.382737 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:52:45.383551 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:45.383575 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:45.384033 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:45.384402 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:45.384828 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:45.386203 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:45.387094 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:45.387818 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:45.387972 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:45.388743 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:45.388749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:45.388765 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:45.389263 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:45.389337 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:45.389368 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:45.389474 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:45.389621 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 06:52:45.389850 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":94,"threads_started":1}
I20260619 06:52:45.391283 4462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c: No bootstrap required, opened a new log
I20260619 06:52:45.393433 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:45.395205 4462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c61a74e85f4af1ae369beab260fb7c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33821 } }
I20260619 06:52:45.395515 4462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:45.395579 4462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c61a74e85f4af1ae369beab260fb7c, State: Initialized, Role: FOLLOWER
I20260619 06:52:45.396066 4462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c61a74e85f4af1ae369beab260fb7c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33821 } }
I20260619 06:52:45.396221 4462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:45.396292 4462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:45.396411 4462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:45.397300 4462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c61a74e85f4af1ae369beab260fb7c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33821 } }
I20260619 06:52:45.397639 4462 leader_election.cc:302] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75c61a74e85f4af1ae369beab260fb7c; no voters:
I20260619 06:52:45.397970 4462 leader_election.cc:288] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:45.398033 4472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:45.398236 4472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [term 1 LEADER]: Becoming Leader. State: Replica: 75c61a74e85f4af1ae369beab260fb7c, State: Running, Role: LEADER
I20260619 06:52:45.398578 4472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c61a74e85f4af1ae369beab260fb7c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33821 } }
I20260619 06:52:45.399241 4462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:45.399415 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:45.395352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56253 (local address 127.14.116.254:33821)
0619 06:52:45.395470 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:45.395474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:45.395638 (+ 164us) server_negotiation.cc:407] Connection header received
0619 06:52:45.395757 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:45.395760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:45.395800 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:45.395894 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:45.396147 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:45.396799 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:45.397644 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:45.397962 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:45.398088 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:45.398967 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:45.398970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:45.398973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:45.398984 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:45.399008 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:45.399019 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:45.399080 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:45.399219 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 06:52:45.399265 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:52:45.400250 4473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75c61a74e85f4af1ae369beab260fb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c61a74e85f4af1ae369beab260fb7c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33821 } } }
I20260619 06:52:45.400538 4474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75c61a74e85f4af1ae369beab260fb7c. Latest consensus state: current_term: 1 leader_uuid: "75c61a74e85f4af1ae369beab260fb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c61a74e85f4af1ae369beab260fb7c" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 33821 } } }
I20260619 06:52:45.400648 4473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:45.400681 4474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:45.401124 4483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:45.403784 4483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:45.409468 4483 catalog_manager.cc:1347] Generated new cluster ID: 861795ff5ea847ccaff3365ce9c64179
I20260619 06:52:45.409565 4483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:45.431342 4483 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:45.432464 4483 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:45.437429 4483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75c61a74e85f4af1ae369beab260fb7c: Generated new TSK 0
I20260619 06:52:45.438100 4483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:45.443981 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 4394
2026-06-19T06:52:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:45 dist-test-slave-9x3b krb5kdc[4493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:45 dist-test-slave-9x3b krb5kdc[4493](info): set up 2 sockets
Jun 19 06:52:45 dist-test-slave-9x3b krb5kdc[4493](info): commencing operation
krb5kdc: starting...
W20260619 06:52:46.475853 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:46 dist-test-slave-9x3b krb5kdc[4493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:46.594664 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:35819
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:35819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:46.702865 4506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:46.703150 4506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:46.703230 4506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:46.707566 4506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:46.707676 4506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:46.707720 4506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:46.707775 4506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:46.707813 4506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:46.713277 4506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:35819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:35819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:46.714329 4506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:46.715325 4506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:46.723591 4511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:46.723585 4514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:46.723843 4506 server_base.cc:1034] running on GCE node
W20260619 06:52:46.723613 4512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:46.724525 4506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:46.725584 4506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:46.726740 4506 hybrid_clock.cc:648] HybridClock initialized: now 1781851966726706 us; error 48 us; skew 500 ppm
Jun 19 06:52:46 dist-test-slave-9x3b krb5kdc[4493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851966, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:46.729398 4506 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:46.730170 4506 webserver.cc:460] Webserver started at http://127.14.116.254:44117/ using document root <none> and password file <none>
I20260619 06:52:46.730654 4506 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:46.730710 4506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:46.730872 4506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:46.732843 4506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "ba4205a0d6fd4c7cb1a8872590798249"
format_stamp: "Formatted at 2026-06-19 06:52:46 on dist-test-slave-9x3b"
server_key: "a5e36ba204d6b8865032006f50fe2966"
server_key_iv: "3c0c8c12379bc72958cdcff2d0e28e92"
server_key_version: "encryptionkey@0"
I20260619 06:52:46.733325 4506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "ba4205a0d6fd4c7cb1a8872590798249"
format_stamp: "Formatted at 2026-06-19 06:52:46 on dist-test-slave-9x3b"
server_key: "a5e36ba204d6b8865032006f50fe2966"
server_key_iv: "3c0c8c12379bc72958cdcff2d0e28e92"
server_key_version: "encryptionkey@0"
I20260619 06:52:46.736793 4506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:52:46.739022 4521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:46.740446 4506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:46.740597 4506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "ba4205a0d6fd4c7cb1a8872590798249"
format_stamp: "Formatted at 2026-06-19 06:52:46 on dist-test-slave-9x3b"
server_key: "a5e36ba204d6b8865032006f50fe2966"
server_key_iv: "3c0c8c12379bc72958cdcff2d0e28e92"
server_key_version: "encryptionkey@0"
I20260619 06:52:46.740706 4506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:46.745131 4506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:46.748114 4506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:46.748294 4506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:46.755086 4506 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:35819
I20260619 06:52:46.755111 4573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:35819 every 8 connection(s)
I20260619 06:52:46.756019 4506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:46.758705 4574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:46.760645 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 4506
I20260619 06:52:46.760789 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:46.761067 14803 external_mini_cluster.cc:1428] Setting key 8fc941882efc92ac7a182a457ad4034c
I20260619 06:52:46.765189 4574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249: Bootstrap starting.
I20260619 06:52:46.766150 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:46.762400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58815 (local address 127.14.116.254:35819)
0619 06:52:46.762851 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:46.762873 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:46.762909 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:52:46.763585 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:46.763602 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:46.763929 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:46.764251 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:46.765030 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:46.765034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:46.765046 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:46.765055 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:46.765106 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:46.765125 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:46.765270 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:46.765341 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 06:52:46.765484 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":125,"threads_started":1}
I20260619 06:52:46.767571 4574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:46.768448 4574 log.cc:824] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:46.769019 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:46.770370 4574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249: No bootstrap required, opened a new log
I20260619 06:52:46.772558 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:46.771026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46243 (local address 127.14.116.254:35819)
0619 06:52:46.771118 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:46.771121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:46.771349 (+ 228us) server_negotiation.cc:407] Connection header received
0619 06:52:46.771462 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:46.771464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:46.771491 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:46.771558 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:46.772224 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:46.772227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:46.772229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:52:46.772239 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:46.772261 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:52:46.772276 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:46.772347 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:46.772447 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 06:52:46.772475 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 06:52:46.774807 4574 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4205a0d6fd4c7cb1a8872590798249" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35819 } }
I20260619 06:52:46.775089 4574 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:52:46.775154 4574 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4205a0d6fd4c7cb1a8872590798249, State: Initialized, Role: FOLLOWER
I20260619 06:52:46.775555 4574 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4205a0d6fd4c7cb1a8872590798249" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35819 } }
I20260619 06:52:46.775759 4574 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:52:46.775847 4574 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:52:46.775944 4574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:46.776799 4574 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4205a0d6fd4c7cb1a8872590798249" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35819 } }
I20260619 06:52:46.777120 4574 leader_election.cc:302] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba4205a0d6fd4c7cb1a8872590798249; no voters:
I20260619 06:52:46.777441 4574 leader_election.cc:288] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:52:46.777523 4584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:46.777782 4584 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [term 1 LEADER]: Becoming Leader. State: Replica: ba4205a0d6fd4c7cb1a8872590798249, State: Running, Role: LEADER
I20260619 06:52:46.778103 4584 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4205a0d6fd4c7cb1a8872590798249" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35819 } }
I20260619 06:52:46.778650 4574 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:46.779778 4586 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4205a0d6fd4c7cb1a8872590798249. Latest consensus state: current_term: 1 leader_uuid: "ba4205a0d6fd4c7cb1a8872590798249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4205a0d6fd4c7cb1a8872590798249" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35819 } } }
I20260619 06:52:46.779915 4586 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:46.780167 4585 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba4205a0d6fd4c7cb1a8872590798249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4205a0d6fd4c7cb1a8872590798249" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 35819 } } }
I20260619 06:52:46.780275 4585 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:46.780557 4593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:46.783069 4593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:46.788035 4593 catalog_manager.cc:1347] Generated new cluster ID: d1efcb9cc93942dfb24fe655ffbea449
I20260619 06:52:46.788097 4593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:46.812148 4593 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:46.812497 4593 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:46.818918 4593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba4205a0d6fd4c7cb1a8872590798249: Generated new TSK 0
I20260619 06:52:46.819636 4593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:52:46.832928 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 4506
2026-06-19T06:52:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1388 ms)
[----------] 18 tests from AuthTokenIssuingTest (25073 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:46 dist-test-slave-9x3b krb5kdc[4605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:46 dist-test-slave-9x3b krb5kdc[4605](info): set up 2 sockets
Jun 19 06:52:46 dist-test-slave-9x3b krb5kdc[4605](info): commencing operation
krb5kdc: starting...
W20260619 06:52:47.859908 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:47 dist-test-slave-9x3b krb5kdc[4605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:47.975905 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:40549
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40549,127.14.116.253:36937,127.14.116.252:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:48.085717 4618 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:52:48.086063 4618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:48.086145 4618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:48.086225 4618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:48.090195 4618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:48.090288 4618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:48.090335 4618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:48.090386 4618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:48.090422 4618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:48.095654 4618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40549,127.14.116.253:36937,127.14.116.252:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:40549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:48.096736 4618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:48.097760 4618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:48.105993 4624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:48.105990 4626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:48.106004 4623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:48.106315 4618 server_base.cc:1034] running on GCE node
I20260619 06:52:48.106896 4618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:48.108007 4618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:48.109172 4618 hybrid_clock.cc:648] HybridClock initialized: now 1781851968109151 us; error 32 us; skew 500 ppm
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:48.111915 4618 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:48.112694 4618 webserver.cc:460] Webserver started at http://127.14.116.254:35313/ using document root <none> and password file <none>
I20260619 06:52:48.113214 4618 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:48.113269 4618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:48.113430 4618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:48.115391 4618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "e2b2d567c2be4ffdba09914f592b89d6"
format_stamp: "Formatted at 2026-06-19 06:52:48 on dist-test-slave-9x3b"
server_key: "484179eb571ba22e643239d122c26ebc"
server_key_iv: "c3a9f98c9cc884207c530c8f677fcbf7"
server_key_version: "encryptionkey@0"
I20260619 06:52:48.115975 4618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "e2b2d567c2be4ffdba09914f592b89d6"
format_stamp: "Formatted at 2026-06-19 06:52:48 on dist-test-slave-9x3b"
server_key: "484179eb571ba22e643239d122c26ebc"
server_key_iv: "c3a9f98c9cc884207c530c8f677fcbf7"
server_key_version: "encryptionkey@0"
I20260619 06:52:48.119477 4618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:52:48.121857 4633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:48.123107 4618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:48.123206 4618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "e2b2d567c2be4ffdba09914f592b89d6"
format_stamp: "Formatted at 2026-06-19 06:52:48 on dist-test-slave-9x3b"
server_key: "484179eb571ba22e643239d122c26ebc"
server_key_iv: "c3a9f98c9cc884207c530c8f677fcbf7"
server_key_version: "encryptionkey@0"
I20260619 06:52:48.123292 4618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:48.142954 4618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:48.145725 4618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:48.145877 4618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:48.152673 4618 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:40549
I20260619 06:52:48.153009 4685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:40549 every 8 connection(s)
I20260619 06:52:48.153641 4618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:48.156329 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:48.160060 4686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:48.161612 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 4618
I20260619 06:52:48.161741 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:48.162024 14803 external_mini_cluster.cc:1428] Setting key 626b53c17d3188044e1813fb08e84496
WARNING: no policy specified for kudu/127.14.116.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.253@KRBTEST.COM" created.
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:48.178966 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.163139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.14.116.254:40549)
0619 06:52:48.163481 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.163495 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.163549 (+ 54us) server_negotiation.cc:407] Connection header received
0619 06:52:48.164324 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.164331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.164511 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.164676 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.165793 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.166735 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.168362 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.168723 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.168848 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.173782 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.173792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.173809 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.173841 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.175748 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.176357 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.176362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.176363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.176468 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.176857 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.176860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.176862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.177246 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.177369 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.177880 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.178159 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260619 06:52:48.178969 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.162260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40549 (local address 127.0.0.1:41066)
0619 06:52:48.162736 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.162797 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:48.164002 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:48.164736 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:48.164744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:48.165253 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:48.165634 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.165654 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.166872 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.166879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.168226 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.168242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.168896 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.168901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.169041 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.169643 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:48.169668 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:48.173615 (+ 3947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:48.175860 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.175872 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.175886 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.176224 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.176567 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.176573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.176578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.176736 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.177384 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:48.177395 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:48.177576 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 06:52:48.177790 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 06:52:48.178260 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":87,"thread_start_us":140,"threads_started":1}
I20260619 06:52:48.181324 4686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:48.182022 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.181795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:36937 (local address 127.0.0.1:56488)
0619 06:52:48.181906 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.181970 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:52:48.182224 4636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
I20260619 06:52:48.183079 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.182889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:36937 (local address 127.0.0.1:56498)
0619 06:52:48.182990 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.183033 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 06:52:48.183265 4686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.253:36937: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab.
Entry for principal kudu/127.14.116.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab.
I20260619 06:52:48.188305 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.253:36937
--webserver_interface=127.14.116.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab
--principal=kudu/127.14.116.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40549,127.14.116.253:36937,127.14.116.252:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:52:48.232528 4686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } attempt: 1
I20260619 06:52:48.233407 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.233114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:36937 (local address 127.0.0.1:56510)
0619 06:52:48.233266 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.233330 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:52:48.234124 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.233918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:36937 (local address 127.0.0.1:56514)
0619 06:52:48.234025 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.234075 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 06:52:48.234264 4686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.253:36937: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
W20260619 06:52:48.297217 4693 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:52:48.297547 4693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:48.297637 4693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:48.297745 4693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:48.301844 4693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:48.301936 4693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:48.301975 4693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:48.302030 4693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:48.302066 4693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:48.302434 4686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } attempt: 2
I20260619 06:52:48.303251 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.302985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:36937 (local address 127.0.0.1:56528)
0619 06:52:48.303116 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.303175 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 06:52:48.303999 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.303785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:36937 (local address 127.0.0.1:56538)
0619 06:52:48.303897 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.303948 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:52:48.304153 4686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.253:36937: Network error: Client connection negotiation failed: client connection to 127.14.116.253:36937: connect: Connection refused (error 111)
I20260619 06:52:48.307822 4693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40549,127.14.116.253:36937,127.14.116.252:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab
--principal=kudu/127.14.116.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.253:36937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.116.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:48.308866 4693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:48.309870 4693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:48.318099 4698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:48.318132 4699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:48.318109 4701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:48.318360 4693 server_base.cc:1034] running on GCE node
I20260619 06:52:48.319083 4693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:48.320155 4693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:48.321321 4693 hybrid_clock.cc:648] HybridClock initialized: now 1781851968321306 us; error 26 us; skew 500 ppm
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:48.323927 4693 init.cc:377] Logged in from keytab as kudu/127.14.116.253@KRBTEST.COM (short username kudu)
I20260619 06:52:48.324704 4693 webserver.cc:460] Webserver started at http://127.14.116.253:36677/ using document root <none> and password file <none>
I20260619 06:52:48.325198 4693 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:48.325254 4693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:48.325408 4693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:48.327314 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/data/instance:
uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07"
format_stamp: "Formatted at 2026-06-19 06:52:48 on dist-test-slave-9x3b"
server_key: "fd5ffaa1e04f8c54c61821daf9762186"
server_key_iv: "26015050cf0c4a0daa0fe883e5663ca1"
server_key_version: "encryptionkey@0"
I20260619 06:52:48.327891 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/wal/instance:
uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07"
format_stamp: "Formatted at 2026-06-19 06:52:48 on dist-test-slave-9x3b"
server_key: "fd5ffaa1e04f8c54c61821daf9762186"
server_key_iv: "26015050cf0c4a0daa0fe883e5663ca1"
server_key_version: "encryptionkey@0"
I20260619 06:52:48.331449 4693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:52:48.333844 4708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:48.335107 4693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:48.335245 4693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/wal
uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07"
format_stamp: "Formatted at 2026-06-19 06:52:48 on dist-test-slave-9x3b"
server_key: "fd5ffaa1e04f8c54c61821daf9762186"
server_key_iv: "26015050cf0c4a0daa0fe883e5663ca1"
server_key_version: "encryptionkey@0"
I20260619 06:52:48.335357 4693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:48.352599 4693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:48.355729 4693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:48.355938 4693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:48.362973 4693 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.253:36937
I20260619 06:52:48.363265 4760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.253:36937 every 8 connection(s)
I20260619 06:52:48.363981 4693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
I20260619 06:52:48.366672 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:48.370349 4761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:48.374117 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 4693
I20260619 06:52:48.374250 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-1/wal/instance
I20260619 06:52:48.374572 14803 external_mini_cluster.cc:1428] Setting key d775d08bca65a67eec320bf0d35c0bac
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.116.252@KRBTEST.COM; defaulting to no policy
I20260619 06:52:48.385109 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.372169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.14.116.254:40549)
0619 06:52:48.372311 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.372315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.373094 (+ 779us) server_negotiation.cc:407] Connection header received
0619 06:52:48.373967 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.373972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.374031 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.374135 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.375664 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.376435 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.378205 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.378437 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.378517 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.381426 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.381429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.381431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.381456 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.382947 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.383474 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.383478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.383480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.383564 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.383951 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.383954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.383955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.384142 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.384229 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.384811 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.384985 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.252@KRBTEST.COM" created.
I20260619 06:52:48.386042 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.372499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40549 (local address 127.0.0.1:41076)
0619 06:52:48.372930 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.372981 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:48.373742 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:48.374393 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:48.374401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:48.374784 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:48.375442 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.375460 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.376566 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.376573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.378070 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.378084 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.378604 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.378609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.378852 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.379487 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:48.379512 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:48.381241 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:48.383071 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.383077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.383093 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.383364 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.383675 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.383680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.383682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.383858 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.384245 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:48.384251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:48.384512 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 06:52:48.384677 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 06:52:48.385020 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20260619 06:52:48.387753 4761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
I20260619 06:52:48.395341 4686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } attempt: 3
I20260619 06:52:48.397320 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.388290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:36937 (local address 127.0.0.1:56542)
0619 06:52:48.388428 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.388442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:48.388539 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:48.389001 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:48.389004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:48.389171 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:48.389250 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.389255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.390187 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.390193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.390722 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.390729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.391326 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.391329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.391511 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.392012 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:48.392025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:48.393378 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:48.395279 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.395284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.395289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.395513 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.396122 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.396127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.396129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.396220 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.396730 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:48.396733 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:48.396832 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 06:52:48.396949 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 06:52:48.397125 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":61}
I20260619 06:52:48.397457 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.388398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.14.116.253:36937)
0619 06:52:48.388683 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.388692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.388709 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:52:48.388739 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.388742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.388887 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.389025 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.389359 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.390076 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.390853 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.391213 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.391439 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.393506 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.393515 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.393535 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.393554 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.395174 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.395674 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.395679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.395680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.395777 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.396324 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.396327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.396328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.396624 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.396732 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.397055 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.397232 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260619 06:52:48.399380 4761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab.
Entry for principal kudu/127.14.116.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab.
I20260619 06:52:48.400182 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.399860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34112)
0619 06:52:48.400013 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.400115 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 06:52:48.400738 4709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
I20260619 06:52:48.401109 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.252:39595
--webserver_interface=127.14.116.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab
--principal=kudu/127.14.116.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:46767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:40549,127.14.116.253:36937,127.14.116.252:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:52:48.401577 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.401365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34126)
0619 06:52:48.401464 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.401513 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
W20260619 06:52:48.402235 4761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:39595: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
I20260619 06:52:48.405740 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.395794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:36937 (local address 127.0.0.1:56554)
0619 06:52:48.395922 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.395938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:48.396017 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:48.396491 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:48.396494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:48.396658 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:48.397244 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.397255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.398392 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.398396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.399194 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.399207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.399901 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.399905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.400017 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.400550 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:48.400564 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:48.402102 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:48.404198 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.404205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.404210 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.404481 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.404743 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.404749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.404754 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.404867 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.405220 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:48.405224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:48.405332 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 06:52:48.405443 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 06:52:48.405588 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 06:52:48.405759 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.395871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.14.116.253:36937)
0619 06:52:48.396262 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.396276 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.396289 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:52:48.396332 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.396335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.396379 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.396467 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.397366 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.398245 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.399314 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.399755 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.399892 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.402242 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.402245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.402247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.402266 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.404088 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.404584 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.404586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.404587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.404657 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.404960 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.404962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.404963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.405116 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.405228 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.405493 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.405597 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":182,"threads_started":1}
I20260619 06:52:48.406445 4686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:48.407160 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.406933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34132)
0619 06:52:48.407044 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.407094 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 06:52:48.407915 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.407588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34146)
0619 06:52:48.407803 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.407855 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260619 06:52:48.408126 4686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:39595: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
I20260619 06:52:48.439285 4686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } attempt: 1
I20260619 06:52:48.440261 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.439882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34160)
0619 06:52:48.440119 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.440184 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 06:52:48.440985 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.440771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34174)
0619 06:52:48.440883 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.440936 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:52:48.441114 4686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:39595: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
I20260619 06:52:48.451493 4761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } attempt: 1
I20260619 06:52:48.452216 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.451962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34180)
0619 06:52:48.452090 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.452151 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 06:52:48.452950 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.452730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34194)
0619 06:52:48.452841 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.452888 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:52:48.453135 4761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:39595: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
W20260619 06:52:48.511509 4769 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:52:48.511894 4769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:48.511966 4769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:48.512038 4769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 06:52:48.516265 4686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } attempt: 2
W20260619 06:52:48.516528 4769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:48.516605 4769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:48.516649 4769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:48.516705 4769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:48.516741 4769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:48.517128 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.516851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34202)
0619 06:52:48.516985 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.517052 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 06:52:48.517843 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.517623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34214)
0619 06:52:48.517733 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.517779 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:52:48.517985 4686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:39595: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
I20260619 06:52:48.521319 4761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } attempt: 2
I20260619 06:52:48.521955 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.521707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34226)
0619 06:52:48.521840 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.521895 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 06:52:48.522607 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.522417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34230)
0619 06:52:48.522516 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.522558 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 06:52:48.522671 4769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:46767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:40549,127.14.116.253:36937,127.14.116.252:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab
--principal=kudu/127.14.116.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.252:39595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.116.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
W20260619 06:52:48.522820 4761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:39595: Network error: Client connection negotiation failed: client connection to 127.14.116.252:39595: connect: Connection refused (error 111)
I20260619 06:52:48.523792 4769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:48.524755 4769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:48.533129 4775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:48.533174 4777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:48.533174 4774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:48.533454 4769 server_base.cc:1034] running on GCE node
I20260619 06:52:48.534047 4769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:48.535130 4769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:48.536317 4769 hybrid_clock.cc:648] HybridClock initialized: now 1781851968536306 us; error 34 us; skew 500 ppm
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:48.539110 4769 init.cc:377] Logged in from keytab as kudu/127.14.116.252@KRBTEST.COM (short username kudu)
I20260619 06:52:48.540055 4769 webserver.cc:460] Webserver started at http://127.14.116.252:40269/ using document root <none> and password file <none>
I20260619 06:52:48.540606 4769 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:48.540695 4769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:48.540922 4769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:48.542906 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/data/instance:
uuid: "3fdcac706c154c40a9c28bcceb020497"
format_stamp: "Formatted at 2026-06-19 06:52:48 on dist-test-slave-9x3b"
server_key: "cf99d6f1c8736f7c41395bb91d17c084"
server_key_iv: "0c88a681127773b126e161b65fc660c3"
server_key_version: "encryptionkey@0"
I20260619 06:52:48.543478 4769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/wal/instance:
uuid: "3fdcac706c154c40a9c28bcceb020497"
format_stamp: "Formatted at 2026-06-19 06:52:48 on dist-test-slave-9x3b"
server_key: "cf99d6f1c8736f7c41395bb91d17c084"
server_key_iv: "0c88a681127773b126e161b65fc660c3"
server_key_version: "encryptionkey@0"
I20260619 06:52:48.547132 4769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:52:48.549917 4784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:48.551196 4769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:52:48.551338 4769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/wal
uuid: "3fdcac706c154c40a9c28bcceb020497"
format_stamp: "Formatted at 2026-06-19 06:52:48 on dist-test-slave-9x3b"
server_key: "cf99d6f1c8736f7c41395bb91d17c084"
server_key_iv: "0c88a681127773b126e161b65fc660c3"
server_key_version: "encryptionkey@0"
I20260619 06:52:48.551453 4769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:48.583025 4769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:48.586018 4769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:48.586225 4769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:48.593092 4769 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.252:39595
I20260619 06:52:48.593394 4836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.252:39595 every 8 connection(s)
I20260619 06:52:48.594049 4769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
I20260619 06:52:48.596808 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:48.598362 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 4769
I20260619 06:52:48.598492 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781851695409368-14803-0/minicluster-data/master-2/wal/instance
I20260619 06:52:48.598788 14803 external_mini_cluster.cc:1428] Setting key e5b3fcdbe25945566b137193373deaae
I20260619 06:52:48.601078 4837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:48.612396 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.603044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.14.116.254:40549)
0619 06:52:48.603170 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.603176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.603450 (+ 274us) server_negotiation.cc:407] Connection header received
0619 06:52:48.603606 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.603645 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.603702 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.603791 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.604516 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.605026 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.606089 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.606397 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.606481 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.609012 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.609015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.609017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.609039 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.610456 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.610982 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.610987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.610989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.611071 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.611386 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.611390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.611392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.611583 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.611713 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.612091 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.612210 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:52:48.613212 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.602964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:40549 (local address 127.0.0.1:41078)
0619 06:52:48.603332 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.603369 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:48.603506 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:48.603842 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:48.603847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:48.604242 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:48.604391 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.604399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.605145 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.605148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.605934 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.605948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.606548 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.606552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.606670 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.607167 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:48.607186 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:48.608865 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:48.610616 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.610624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.610633 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.610878 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.611164 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.611169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.611171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.611287 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.611724 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:48.611732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:48.611868 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 06:52:48.612036 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 06:52:48.612371 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260619 06:52:48.613214 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.600089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.14.116.252:39595)
0619 06:52:48.600465 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.600489 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.600536 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:52:48.601152 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.601167 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.601542 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.601815 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.602655 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.604032 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.604983 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.605513 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.605622 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.608611 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.608617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.608627 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.608648 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.610398 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.610834 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.610840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.610842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.610974 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.611276 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.611280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.611282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.611672 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.611879 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.612108 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.612458 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":247,"spinlock_wait_cycles":9088,"thread_start_us":138,"threads_started":1}
I20260619 06:52:48.613986 4761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } attempt: 3
I20260619 06:52:48.614861 4837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:48.617147 4686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } attempt: 3
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
I20260619 06:52:48.624393 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.614461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34236)
0619 06:52:48.614639 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.614654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:48.614732 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:48.615453 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:48.615456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:48.615700 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:48.615810 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.615818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.616994 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.616997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.617788 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.617799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.618241 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.618244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.618336 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.618806 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:48.618820 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:48.620276 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:48.622350 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.622356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.622359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.622599 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.622935 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.622940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.622945 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.623050 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.623803 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:48.623809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:48.623917 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 06:52:48.624027 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 06:52:48.624179 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 06:52:48.625100 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.614490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.14.116.252:39595)
0619 06:52:48.614962 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.614970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.614983 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:52:48.615014 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.615016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.615055 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.615161 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.615933 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.616885 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.617915 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.618128 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.618240 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.620429 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.620433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.620435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.620459 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.622226 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.622723 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.622726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.622729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.622826 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.623172 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.623177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.623179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.623702 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.623800 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.624729 (+ 929us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.624912 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":114,"threads_started":1}
I20260619 06:52:48.627449 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.615993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.14.116.254:40549)
0619 06:52:48.616091 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.616096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.616109 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:52:48.616146 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.616148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.616197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.616318 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.617948 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.618711 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.619815 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.620132 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.620198 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.623666 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.623670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.623672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.623700 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.625385 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.625932 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.625938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.625943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.626026 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.626358 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.626361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.626363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.626565 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.626814 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.627165 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.627269 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 06:52:48.627964 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.615311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:36937 (local address 127.0.0.1:56568)
0619 06:52:48.615418 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.615432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:48.615495 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:48.616417 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:48.616420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:48.616554 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:48.616635 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.616639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.617603 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.617606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.618888 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.618897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.619676 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.619683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.619788 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.620312 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:48.620330 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:48.623018 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:48.626327 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.626333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.626338 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.626565 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.626883 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.626886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.626888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.626974 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.627461 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:48.627467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:48.627565 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 06:52:48.627695 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 06:52:48.627820 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:52:48.628157 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.617503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34238)
0619 06:52:48.617648 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.617658 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:48.617712 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:48.619235 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:48.619238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:48.619418 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:48.619498 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.619503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.620569 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.620576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.621455 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.621467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.622060 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.622063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.622143 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.622593 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:48.622607 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:48.624043 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:48.626385 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.626388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.626390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.626651 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.627015 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.627018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.627020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.627099 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.627727 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:48.627731 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:48.627828 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 06:52:48.627913 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:52:48.628026 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 06:52:48.628429 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.616051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.14.116.253:36937)
0619 06:52:48.616158 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.616162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.616174 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:52:48.616204 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.616207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.616249 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.616318 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.616749 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.617484 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.618999 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.619425 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.619522 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.623142 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.623145 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.623148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.623172 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.626211 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.626674 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.626678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.626679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.626764 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.627080 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.627085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.627087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.627329 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.627416 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.628172 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.628283 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 06:52:48.628763 4761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07: Bootstrap starting.
I20260619 06:52:48.628927 4837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:48.629575 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.617540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.14.116.252:39595)
0619 06:52:48.619044 (+ 1504us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.619048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.619057 (+ 9us) server_negotiation.cc:407] Connection header received
0619 06:52:48.619083 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.619086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.619121 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.619597 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.619719 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.620442 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.621611 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.621945 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.622059 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.624187 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.624190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.624193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.624216 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.626242 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.626835 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.626838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.626839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.626916 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.627205 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.627208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.627210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.627407 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.628931 (+ 1524us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.629006 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.629165 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1448}
I20260619 06:52:48.631511 4761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:48.632334 4761 log.cc:824] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:48.632357 4686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6: Bootstrap starting.
I20260619 06:52:48.634303 4761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07: No bootstrap required, opened a new log
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
I20260619 06:52:48.634981 4686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:48.635843 4686 log.cc:824] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:48.637408 4686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6: No bootstrap required, opened a new log
I20260619 06:52:48.638559 4761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } }
I20260619 06:52:48.638768 4761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a4b5d8fd644ec8af364f5d1bdd4a07, State: Initialized, Role: FOLLOWER
I20260619 06:52:48.639135 4761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } }
I20260619 06:52:48.639339 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.629489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.14.116.253:36937)
0619 06:52:48.629605 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.629609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.629620 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:52:48.629648 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.629651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.629691 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.629773 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.630484 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.631201 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.632265 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.632592 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.632703 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.635487 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.635490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.635493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.635516 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.637287 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.637751 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.637755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.637757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.637837 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.638119 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.638123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.638125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.638326 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.638509 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.639064 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.639173 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:52:48.640436 4761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:48.642423 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Jun 19 06:52:48 dist-test-slave-9x3b krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851968, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
I20260619 06:52:48.643846 4660 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e2b2d567c2be4ffdba09914f592b89d6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41094
I20260619 06:52:48.643493 4845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.644222 4845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:48.644767 4686 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } }
I20260619 06:52:48.644982 4686 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b2d567c2be4ffdba09914f592b89d6, State: Initialized, Role: FOLLOWER
I20260619 06:52:48.645294 4686 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } }
W20260619 06:52:48.645900 4856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:48.645961 4856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:48.646471 4686 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:48.646986 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.629339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:39595 (local address 127.0.0.1:34242)
0619 06:52:48.636220 (+ 6881us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:48.636238 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:48.636323 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:48.637691 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:48.637694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:48.637874 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:48.637959 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.637965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.639036 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.639038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.640121 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:48.640132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.640803 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.640808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:48.640881 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.641338 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:48.641355 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:48.642925 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:48.645560 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.645562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.645565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.645795 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.646044 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:48.646046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:48.646048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:48.646115 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:48.646504 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:48.646508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:48.646577 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 06:52:48.646662 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:52:48.646812 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6805}
I20260619 06:52:48.647468 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.629399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.14.116.252:39595)
0619 06:52:48.629901 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.629905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.637400 (+ 7495us) server_negotiation.cc:407] Connection header received
0619 06:52:48.637439 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.637442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.637484 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.637551 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.638073 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.638925 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.640241 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.640593 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.640685 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.643053 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.643057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.643063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.643088 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.645451 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.645891 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.645894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.645895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.645964 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.646195 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.646198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.646200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.646387 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.646474 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.647221 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.647331 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451}
W20260619 06:52:48.647820 4639 tablet.cc:2369] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:52:48.647987 4858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.648102 4858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:48.648327 4660 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:52:48.645046 4693 master_runner.cc:186] Error getting master registration for 127.14.116.254:40549: OK, instance_id { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" instance_seqno: 1781851968150678 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:52:48.648443 4660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:48.648559 4693 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:52:48.649636 4660 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } }
I20260619 06:52:48.650388 4837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497: Bootstrap starting.
I20260619 06:52:48.650518 4660 leader_election.cc:288] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [CANDIDATE]: Term 1 election: Requested vote from peers 45a4b5d8fd644ec8af364f5d1bdd4a07 (127.14.116.253:36937), 3fdcac706c154c40a9c28bcceb020497 (127.14.116.252:39595)
I20260619 06:52:48.651017 4735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2b2d567c2be4ffdba09914f592b89d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07"
I20260619 06:52:48.651249 4735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:48.652731 4735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b2d567c2be4ffdba09914f592b89d6 in term 1.
I20260619 06:52:48.652981 4837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:48.653797 4837 log.cc:824] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:48.653854 4636 leader_election.cc:302] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a4b5d8fd644ec8af364f5d1bdd4a07, e2b2d567c2be4ffdba09914f592b89d6; no voters:
I20260619 06:52:48.654884 4858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:48.655138 4858 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [term 1 LEADER]: Becoming Leader. State: Replica: e2b2d567c2be4ffdba09914f592b89d6, State: Running, Role: LEADER
I20260619 06:52:48.655475 4858 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } }
I20260619 06:52:48.656222 4837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497: No bootstrap required, opened a new log
I20260619 06:52:48.657260 4858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2b2d567c2be4ffdba09914f592b89d6. Latest consensus state: current_term: 1 leader_uuid: "e2b2d567c2be4ffdba09914f592b89d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.657375 4858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [sys.catalog]: This master's current role is: LEADER
W20260619 06:52:48.660528 4869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:48.660609 4869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:48.661381 4837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } }
I20260619 06:52:48.661628 4837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fdcac706c154c40a9c28bcceb020497, State: Initialized, Role: FOLLOWER
I20260619 06:52:48.661962 4837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } }
I20260619 06:52:48.663793 4837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:48.663859 4811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2b2d567c2be4ffdba09914f592b89d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3fdcac706c154c40a9c28bcceb020497"
I20260619 06:52:48.664050 4811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:48.665869 4811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b2d567c2be4ffdba09914f592b89d6 in term 1.
I20260619 06:52:48.666013 4882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.666128 4882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:48.666949 4878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:48.667901 4878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:52:48.662986 4618 master_runner.cc:186] Error getting master registration for 127.14.116.252:39595: OK, instance_id { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" instance_seqno: 1781851968591106 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:52:48.668735 4618 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:52:48.673200 4735 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [term 1 FOLLOWER]: Refusing update from remote peer e2b2d567c2be4ffdba09914f592b89d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:48.673789 4858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:48.674719 4811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [term 1 FOLLOWER]: Refusing update from remote peer e2b2d567c2be4ffdba09914f592b89d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:48.675176 4858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:48.677201 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.654380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.14.116.253:36937)
0619 06:52:48.654491 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.654496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.662130 (+ 7634us) server_negotiation.cc:407] Connection header received
0619 06:52:48.662506 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.662509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.662555 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.662606 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.666575 (+ 3969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.667299 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.669313 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.669620 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.669708 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.673236 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.673240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.673242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.673268 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.675223 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.675841 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.675845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.675847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.675931 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.676231 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.676234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.676236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.676462 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.676573 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.676921 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.677039 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:52:48.678761 4845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2b2d567c2be4ffdba09914f592b89d6. Latest consensus state: current_term: 1 leader_uuid: "e2b2d567c2be4ffdba09914f592b89d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.678882 4845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 06:52:48.680998 4893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:48.681072 4893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:48.682624 4882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2b2d567c2be4ffdba09914f592b89d6. Latest consensus state: current_term: 1 leader_uuid: "e2b2d567c2be4ffdba09914f592b89d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.682739 4882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:48.686903 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.660152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.14.116.254:40549)
0619 06:52:48.660254 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.660259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.677998 (+ 17739us) server_negotiation.cc:407] Connection header received
0619 06:52:48.678167 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.678171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.678217 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.678267 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.679236 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.680000 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.681022 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.681357 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.681435 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.683369 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.683373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.683375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.683403 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.685112 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.685595 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.685599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.685601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.685681 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.685973 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.685977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.685978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.686179 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.686293 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.686642 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.686754 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 06:52:48.690097 4858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2b2d567c2be4ffdba09914f592b89d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.690218 4858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:48.690305 4858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2b2d567c2be4ffdba09914f592b89d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.690378 4858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6 [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:48.690974 4882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2b2d567c2be4ffdba09914f592b89d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.691092 4882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:48.691439 4898 mvcc.cc:205] Tried to move back new op lower bound from 7298465663677157376 to 7298465663616557056. Current Snapshot: MvccSnapshot[applied={T|T < 7298465663677157376}]
I20260619 06:52:48.691479 4845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2b2d567c2be4ffdba09914f592b89d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2d567c2be4ffdba09914f592b89d6" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 40549 } } peers { permanent_uuid: "45a4b5d8fd644ec8af364f5d1bdd4a07" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 36937 } } peers { permanent_uuid: "3fdcac706c154c40a9c28bcceb020497" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 39595 } } }
I20260619 06:52:48.691628 4845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:48.691960 4894 mvcc.cc:205] Tried to move back new op lower bound from 7298465663677157376 to 7298465663616557056. Current Snapshot: MvccSnapshot[applied={T|T < 7298465663677157376}]
I20260619 06:52:48.692240 4896 mvcc.cc:205] Tried to move back new op lower bound from 7298465663677157376 to 7298465663616557056. Current Snapshot: MvccSnapshot[applied={T|T < 7298465663677157376}]
I20260619 06:52:48.695315 4878 catalog_manager.cc:1347] Generated new cluster ID: c9581a36a8724cb091b66b31cf5c84c1
I20260619 06:52:48.695385 4878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:48.697708 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:48.661133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.14.116.252:39595)
0619 06:52:48.661238 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.661242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.681343 (+ 20101us) server_negotiation.cc:407] Connection header received
0619 06:52:48.681513 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.681519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.681567 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.681616 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.682589 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.683379 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.691696 (+ 8317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:48.692040 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:48.692121 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:48.693949 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:48.693953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:48.693958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:48.693989 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:48.695805 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.696356 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.696360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.696361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.696445 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:48.696750 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:48.696753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:48.696755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:48.696964 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:48.697063 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:48.697434 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 06:52:48.697550 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:52:48.718652 4878 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:48.719945 4878 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:48.737071 4878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2b2d567c2be4ffdba09914f592b89d6: Generated new TSK 0
I20260619 06:52:48.737957 4878 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:52:48.769744 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.766737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.14.116.253:36937)
0619 06:52:48.766873 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.766877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.768463 (+ 1586us) server_negotiation.cc:407] Connection header received
0619 06:52:48.768890 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.768895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.768944 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.768997 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.769607 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56608: BlockingRecv error: recv got EOF from 127.0.0.1:56608 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 06:52:48.771034 4689 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.767528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.14.116.254:40549)
0619 06:52:48.767655 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.767659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.770199 (+ 2540us) server_negotiation.cc:407] Connection header received
0619 06:52:48.770358 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.770361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.770407 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.770457 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.770914 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41106: BlockingRecv error: recv got EOF from 127.0.0.1:41106 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 06:52:48.772334 4840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.768137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.14.116.252:39595)
0619 06:52:48.768241 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.768245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.771384 (+ 3139us) server_negotiation.cc:407] Connection header received
0619 06:52:48.771662 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.771665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.771713 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.771763 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.772223 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34266: BlockingRecv error: recv got EOF from 127.0.0.1:34266 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 06:52:48.778018 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.776536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.14.116.253:36937)
0619 06:52:48.776631 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.776634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.777091 (+ 457us) server_negotiation.cc:407] Connection header received
0619 06:52:48.777295 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.777298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.777376 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.777433 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.777923 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56618: BlockingRecv error: recv got EOF from 127.0.0.1:56618 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 06:52:48.783046 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.781562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.14.116.253:36937)
0619 06:52:48.781665 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.781669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.782185 (+ 516us) server_negotiation.cc:407] Connection header received
0619 06:52:48.782365 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.782369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.782417 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.782467 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.782952 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56630: BlockingRecv error: recv got EOF from 127.0.0.1:56630 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 06:52:48.788453 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.786902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.14.116.253:36937)
0619 06:52:48.786997 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.787001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.787496 (+ 495us) server_negotiation.cc:407] Connection header received
0619 06:52:48.787741 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.787744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.787794 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.787859 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.788366 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56632: BlockingRecv error: recv got EOF from 127.0.0.1:56632 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 06:52:48.795969 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.794399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.14.116.253:36937)
0619 06:52:48.794524 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.794528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.795051 (+ 523us) server_negotiation.cc:407] Connection header received
0619 06:52:48.795227 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.795231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.795276 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.795325 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.795871 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56636: BlockingRecv error: recv got EOF from 127.0.0.1:56636 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 06:52:48.807602 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.805977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.14.116.253:36937)
0619 06:52:48.806082 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.806086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.806641 (+ 555us) server_negotiation.cc:407] Connection header received
0619 06:52:48.806854 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.806858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.806913 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.806972 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.807500 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56642: BlockingRecv error: recv got EOF from 127.0.0.1:56642 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 06:52:48.827521 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.825766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.14.116.253:36937)
0619 06:52:48.825891 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.825896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.826544 (+ 648us) server_negotiation.cc:407] Connection header received
0619 06:52:48.826732 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.826735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.826786 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.826847 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.827389 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56656: BlockingRecv error: recv got EOF from 127.0.0.1:56656 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 06:52:48.863057 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.861749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.14.116.253:36937)
0619 06:52:48.861888 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.861893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.862149 (+ 256us) server_negotiation.cc:407] Connection header received
0619 06:52:48.862340 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.862343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.862390 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.862480 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.862938 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56672: BlockingRecv error: recv got EOF from 127.0.0.1:56672 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 06:52:48.930330 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:48.929215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.14.116.253:36937)
0619 06:52:48.929411 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:48.929415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:48.929496 (+ 81us) server_negotiation.cc:407] Connection header received
0619 06:52:48.929617 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:48.929620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:48.929666 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:48.929739 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:48.930211 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56680: BlockingRecv error: recv got EOF from 127.0.0.1:56680 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260619 06:52:49.062240 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:49.060640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.14.116.253:36937)
0619 06:52:49.060805 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:49.060819 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:49.061064 (+ 245us) server_negotiation.cc:407] Connection header received
0619 06:52:49.061239 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:49.061243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:49.061304 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:49.061389 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:49.062076 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56686: BlockingRecv error: recv got EOF from 127.0.0.1:56686 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 06:52:49.321153 4766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:49.320124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.14.116.253:36937)
0619 06:52:49.320265 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:49.320268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:49.320309 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:52:49.320458 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:49.320460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:49.320507 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:49.320579 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:49.321014 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56690: BlockingRecv error: recv got EOF from 127.0.0.1:56690 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:52:49.647320 4856 catalog_manager.cc:1259] Loaded cluster ID: c9581a36a8724cb091b66b31cf5c84c1
I20260619 06:52:49.647404 4856 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07: loading cluster ID for follower catalog manager: success
I20260619 06:52:49.649151 4856 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07: acquiring CA information for follower catalog manager: success
I20260619 06:52:49.650646 4856 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45a4b5d8fd644ec8af364f5d1bdd4a07: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:52:49.682351 4893 catalog_manager.cc:1259] Loaded cluster ID: c9581a36a8724cb091b66b31cf5c84c1
I20260619 06:52:49.682452 4893 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497: loading cluster ID for follower catalog manager: success
I20260619 06:52:49.684122 4893 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497: acquiring CA information for follower catalog manager: success
I20260619 06:52:49.685206 4893 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3fdcac706c154c40a9c28bcceb020497: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:52:49.838838 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:49.835196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.14.116.253:36937)
0619 06:52:49.835444 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:49.835448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:49.835465 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:52:49.835576 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:49.835579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:49.835768 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:49.835875 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:49.836092 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.836613 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.837429 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.837647 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.837728 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:49.837998 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:49.838421 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:49.838512 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:49.838591 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 06:52:49.838636 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260619 06:52:49.845250 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:49.841821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.14.116.252:39595)
0619 06:52:49.842055 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:49.842059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:49.842140 (+ 81us) server_negotiation.cc:407] Connection header received
0619 06:52:49.842245 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:49.842247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:49.842356 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:49.842471 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:49.842664 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.843151 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.843969 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.844199 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.844278 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:49.844506 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:49.844865 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:49.844953 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:49.845016 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 06:52:49.845057 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260619 06:52:49.852607 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:49.848283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.14.116.254:40549)
0619 06:52:49.848647 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:49.848651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:49.848665 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:52:49.848796 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:49.848800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:49.848965 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:49.849077 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:49.849425 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.850226 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.851258 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.851495 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.851605 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:49.851994 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:49.852223 (+ 229us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:49.852338 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:49.852408 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 06:52:49.852461 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":65,"threads_started":1}
I20260619 06:52:49.853077 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:49.848639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.14.116.252:39595)
0619 06:52:49.848745 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:49.848748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:49.849111 (+ 363us) server_negotiation.cc:407] Connection header received
0619 06:52:49.849201 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:49.849204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:49.849248 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:49.849331 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:49.849715 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.850211 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.851656 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.851994 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.852068 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:49.852571 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:49.852640 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:49.852686 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:49.852867 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 06:52:49.852920 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:52:49.854249 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:49.848367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.14.116.253:36937)
0619 06:52:49.848507 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:49.848512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:49.848955 (+ 443us) server_negotiation.cc:407] Connection header received
0619 06:52:49.849071 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:49.849076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:49.849130 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:49.849200 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:49.849646 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.850371 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.851656 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:49.851994 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:49.852068 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:49.852571 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:49.852640 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:49.852686 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:49.854053 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0619 06:52:49.854116 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:52:49.858892 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 4618
I20260619 06:52:49.865280 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 4693
I20260619 06:52:49.870924 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 4769
2026-06-19T06:52:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3042 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:52:49 dist-test-slave-9x3b krb5kdc[4987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:52:49 dist-test-slave-9x3b krb5kdc[4987](info): set up 2 sockets
Jun 19 06:52:49 dist-test-slave-9x3b krb5kdc[4987](info): commencing operation
krb5kdc: starting...
W20260619 06:52:50.912086 14803 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/test-user.keytab.
Jun 19 06:52:50 dist-test-slave-9x3b krb5kdc[4987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:52:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:52:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.116.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
Entry for principal kudu/127.14.116.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab.
I20260619 06:52:51.035130 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:42807
--webserver_interface=127.14.116.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:40997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:42807,127.14.116.253:46353,127.14.116.252:33275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:52:51.147078 5000 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:52:51.147382 5000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:51.147408 5000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:51.147491 5000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:51.151592 5000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:51.151662 5000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:51.151682 5000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:51.151700 5000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:51.151715 5000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:51.157097 5000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:42807,127.14.116.253:46353,127.14.116.252:33275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.254.keytab
--principal=kudu/127.14.116.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.254:42807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.116.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:51.158174 5000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:51.159221 5000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:51.167466 5005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:51.167541 5008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:51.167665 5000 server_base.cc:1034] running on GCE node
W20260619 06:52:51.167491 5006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:51.168395 5000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:51.169508 5000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:51.170673 5000 hybrid_clock.cc:648] HybridClock initialized: now 1781851971170644 us; error 42 us; skew 500 ppm
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:51.173473 5000 init.cc:377] Logged in from keytab as kudu/127.14.116.254@KRBTEST.COM (short username kudu)
I20260619 06:52:51.174414 5000 webserver.cc:460] Webserver started at http://127.14.116.254:34883/ using document root <none> and password file <none>
I20260619 06:52:51.174980 5000 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:51.175071 5000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:51.175303 5000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:51.177382 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/data/instance:
uuid: "05159fc0c25e44168cf3b6b90f8e3aae"
format_stamp: "Formatted at 2026-06-19 06:52:51 on dist-test-slave-9x3b"
server_key: "05ca72733f5f6ee5fa42cb93f8c21aa9"
server_key_iv: "cd8293de1b30fc40d0355a69c97471b5"
server_key_version: "encryptionkey@0"
I20260619 06:52:51.177953 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/wal/instance:
uuid: "05159fc0c25e44168cf3b6b90f8e3aae"
format_stamp: "Formatted at 2026-06-19 06:52:51 on dist-test-slave-9x3b"
server_key: "05ca72733f5f6ee5fa42cb93f8c21aa9"
server_key_iv: "cd8293de1b30fc40d0355a69c97471b5"
server_key_version: "encryptionkey@0"
I20260619 06:52:51.181615 5000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:52:51.184002 5015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:51.185150 5000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:51.185297 5000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/wal
uuid: "05159fc0c25e44168cf3b6b90f8e3aae"
format_stamp: "Formatted at 2026-06-19 06:52:51 on dist-test-slave-9x3b"
server_key: "05ca72733f5f6ee5fa42cb93f8c21aa9"
server_key_iv: "cd8293de1b30fc40d0355a69c97471b5"
server_key_version: "encryptionkey@0"
I20260619 06:52:51.185423 5000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:51.206521 5000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:51.209540 5000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:51.209746 5000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:51.216922 5000 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.254:42807
I20260619 06:52:51.217223 5067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.254:42807 every 8 connection(s)
I20260619 06:52:51.217885 5000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/data/info.pb
I20260619 06:52:51.220775 5068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:51.221769 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 5000
I20260619 06:52:51.221891 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-0/wal/instance
I20260619 06:52:51.222154 14803 external_mini_cluster.cc:1428] Setting key 2fe05859157544cfd068e1b9d2e83083
I20260619 06:52:51.224726 5068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.116.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.253@KRBTEST.COM" created.
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:51.240216 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.227055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:42807 (local address 127.0.0.1:60882)
0619 06:52:51.227547 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.227599 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:51.228553 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:51.229208 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:51.229212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:51.229625 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:51.229878 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.229891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.231023 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.231026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.231977 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.231990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.232600 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.232604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.232754 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.233419 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:51.233434 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:51.234972 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:51.237260 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.237268 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.237275 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.237520 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.237936 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.237939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.237940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.238044 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.238856 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:51.238861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:51.238981 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 06:52:51.239120 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 06:52:51.239377 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":3,"thread_start_us":140,"threads_started":1}
I20260619 06:52:51.240221 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.228059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.14.116.254:42807)
0619 06:52:51.228406 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.228416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.228458 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:52:51.228840 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.228845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.229027 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.229212 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.230021 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.230898 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.232096 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.232454 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.232658 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.235118 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.235123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.235133 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.235154 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.237136 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.237651 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.237660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.237664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.237819 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.238149 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.238154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.238157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.238727 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.238881 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.239236 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.239450 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":236,"thread_start_us":96,"threads_started":1}
I20260619 06:52:51.242710 5068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:51.243371 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.243115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:46353 (local address 127.0.0.1:55856)
0619 06:52:51.243225 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.243297 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 06:52:51.243597 5019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
I20260619 06:52:51.244490 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.244272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:46353 (local address 127.0.0.1:55872)
0619 06:52:51.244379 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.244420 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:52:51.244655 5068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.253:46353: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab.
Entry for principal kudu/127.14.116.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab.
I20260619 06:52:51.248070 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.253:46353
--webserver_interface=127.14.116.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab
--principal=kudu/127.14.116.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:40997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:42807,127.14.116.253:46353,127.14.116.252:33275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:52:51.293872 5068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } attempt: 1
I20260619 06:52:51.294785 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.294453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:46353 (local address 127.0.0.1:55874)
0619 06:52:51.294624 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.294694 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:52:51.295527 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.295308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:46353 (local address 127.0.0.1:55876)
0619 06:52:51.295419 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.295471 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:52:51.295692 5068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.253:46353: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
W20260619 06:52:51.357741 5075 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:52:51.358026 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:51.358081 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:51.358160 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:51.362136 5075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:51.362192 5075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:51.362212 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:51.362273 5075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:51.362309 5075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:51.363857 5068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } attempt: 2
I20260619 06:52:51.364647 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.364348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:46353 (local address 127.0.0.1:55880)
0619 06:52:51.364482 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.364572 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 06:52:51.365481 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.365166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:46353 (local address 127.0.0.1:55886)
0619 06:52:51.365353 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.365398 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260619 06:52:51.365661 5068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.253:46353: Network error: Client connection negotiation failed: client connection to 127.14.116.253:46353: connect: Connection refused (error 111)
I20260619 06:52:51.368125 5075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:42807,127.14.116.253:46353,127.14.116.252:33275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.253.keytab
--principal=kudu/127.14.116.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.253:46353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.116.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:51.369168 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:51.370174 5075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:52:51.378388 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:51.378393 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:51.378427 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:51.378638 5075 server_base.cc:1034] running on GCE node
I20260619 06:52:51.379156 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:51.380319 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:51.381477 5075 hybrid_clock.cc:648] HybridClock initialized: now 1781851971381445 us; error 47 us; skew 500 ppm
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:51.384342 5075 init.cc:377] Logged in from keytab as kudu/127.14.116.253@KRBTEST.COM (short username kudu)
I20260619 06:52:51.385231 5075 webserver.cc:460] Webserver started at http://127.14.116.253:34475/ using document root <none> and password file <none>
I20260619 06:52:51.385790 5075 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:51.385857 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:51.386065 5075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:51.388103 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/data/instance:
uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0"
format_stamp: "Formatted at 2026-06-19 06:52:51 on dist-test-slave-9x3b"
server_key: "a45957c85234bdf2378ce2ec7d66a371"
server_key_iv: "6a4621b2073f6465fe2b28bdb7abc783"
server_key_version: "encryptionkey@0"
I20260619 06:52:51.388659 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/wal/instance:
uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0"
format_stamp: "Formatted at 2026-06-19 06:52:51 on dist-test-slave-9x3b"
server_key: "a45957c85234bdf2378ce2ec7d66a371"
server_key_iv: "6a4621b2073f6465fe2b28bdb7abc783"
server_key_version: "encryptionkey@0"
I20260619 06:52:51.392383 5075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:52:51.394737 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:51.396008 5075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:52:51.396131 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/wal
uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0"
format_stamp: "Formatted at 2026-06-19 06:52:51 on dist-test-slave-9x3b"
server_key: "a45957c85234bdf2378ce2ec7d66a371"
server_key_iv: "6a4621b2073f6465fe2b28bdb7abc783"
server_key_version: "encryptionkey@0"
I20260619 06:52:51.396230 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:51.412142 5075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:51.415452 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:51.415679 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:51.422431 5075 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.253:46353
I20260619 06:52:51.422714 5142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.253:46353 every 8 connection(s)
I20260619 06:52:51.423394 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/data/info.pb
I20260619 06:52:51.423956 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 5075
I20260619 06:52:51.424095 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-1/wal/instance
I20260619 06:52:51.424409 14803 external_mini_cluster.cc:1428] Setting key 8e737de2781e97d81da6c8c6574c895b
I20260619 06:52:51.426469 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:51.432051 5143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.116.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.116.252@KRBTEST.COM" created.
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:51.447400 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.434313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.14.116.254:42807)
0619 06:52:51.434464 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.434468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.435312 (+ 844us) server_negotiation.cc:407] Connection header received
0619 06:52:51.436674 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.436677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.436741 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.436851 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.438196 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.439084 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.440364 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.440818 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.440960 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.443743 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.443746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.443748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.443775 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.445272 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.445860 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.445862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.445864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.445931 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.446286 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.446288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.446290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.446478 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.446550 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.447081 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.447181 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:52:51.448336 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.434663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:42807 (local address 127.0.0.1:60898)
0619 06:52:51.435156 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.435199 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:51.436398 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:51.437086 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:51.437098 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:51.437546 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:51.438018 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.438032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.439222 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.439225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.440226 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.440235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.440993 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.440996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.441144 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.441793 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:51.441811 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:51.443537 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:51.445407 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.445413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.445429 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.445764 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.446061 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.446065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.446066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.446198 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.446563 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:51.446568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:51.446848 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 06:52:51.447006 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 06:52:51.447343 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.116.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab.
Entry for principal kudu/127.14.116.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab.
I20260619 06:52:51.449878 5143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:51.450824 14803 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2eB_D1/build/debug/bin/kudu
/tmp/dist-test-task2eB_D1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.252:33275
--webserver_interface=127.14.116.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab
--principal=kudu/127.14.116.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.116.212:40997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.116.254:42807,127.14.116.253:46353,127.14.116.252:33275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
I20260619 06:52:51.456816 5068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } attempt: 3
I20260619 06:52:51.459163 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.450390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:46353 (local address 127.0.0.1:55888)
0619 06:52:51.450500 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.450516 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:51.450595 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:51.451205 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:51.451207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:51.451377 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:51.451458 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.451463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.452461 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.452467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.453010 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.453017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.453605 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.453609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.453699 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.454208 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:51.454218 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:51.455468 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:51.457521 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.457524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.457525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.457707 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.458000 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.458003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.458005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.458099 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.458692 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:51.458695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:51.458798 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 06:52:51.458904 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:52:51.459034 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:52:51.459779 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.450539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.14.116.253:46353)
0619 06:52:51.450809 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.450820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.450838 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:52:51.450870 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.450874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.451080 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.451230 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.451573 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.452346 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.453123 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.453488 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.453610 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.455639 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.455653 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.455669 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.455697 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.457391 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.457816 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.457821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.457823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.457905 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.458194 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.458199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.458204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.458511 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.458652 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.459396 (+ 744us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.459546 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260619 06:52:51.461733 5143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:51.462455 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.462168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48652)
0619 06:52:51.462316 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.462388 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 06:52:51.462823 5093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
I20260619 06:52:51.464061 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.463832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48654)
0619 06:52:51.463975 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.464013 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
W20260619 06:52:51.464608 5143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:33275: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
I20260619 06:52:51.467995 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.457292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:46353 (local address 127.0.0.1:55904)
0619 06:52:51.457518 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.457534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:51.457621 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:51.458156 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:51.458162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:51.458346 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:51.458930 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.458941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.460427 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.460431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.461148 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.461159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.462291 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.462295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.462396 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.463310 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:51.463322 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:51.464859 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:51.466473 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.466476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.466478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.466689 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.466975 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.466978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.466980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.467065 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.467438 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:51.467442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:51.467548 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 06:52:51.467699 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 06:52:51.467844 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 06:52:51.468022 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.457353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.14.116.253:46353)
0619 06:52:51.457892 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.457902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.457917 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:52:51.457959 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.457964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.458016 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.458119 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.459060 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.460288 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.461263 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.462156 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.462611 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.465016 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.465020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.465022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.465050 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.466351 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.466786 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.466791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.466793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.466875 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.467160 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.467163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.467164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.467334 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.467437 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.467751 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.467853 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":119,"threads_started":1}
I20260619 06:52:51.468845 5068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:51.469487 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.469280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48670)
0619 06:52:51.469392 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.469436 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 06:52:51.470147 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.469914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48678)
0619 06:52:51.470044 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.470086 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 06:52:51.470290 5068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:33275: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
I20260619 06:52:51.501425 5068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } attempt: 1
I20260619 06:52:51.502255 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.501967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48680)
0619 06:52:51.502118 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.502178 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:52:51.503024 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.502821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48684)
0619 06:52:51.502926 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.502975 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:52:51.503146 5068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:33275: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
I20260619 06:52:51.513880 5143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } attempt: 1
I20260619 06:52:51.514669 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.514399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48698)
0619 06:52:51.514537 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.514601 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:52:51.515406 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.515214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48700)
0619 06:52:51.515313 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.515356 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:52:51.515648 5143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:33275: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
W20260619 06:52:51.564512 5149 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:52:51.564791 5149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:52:51.564815 5149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:52:51.564913 5149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:52:51.568908 5149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:52:51.568961 5149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:52:51.568981 5149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:52:51.569029 5149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:52:51.569062 5149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:52:51.574292 5149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.116.212:40997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.116.254:42807,127.14.116.253:46353,127.14.116.252:33275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/krb5kdc/kudu_127.14.116.252.keytab
--principal=kudu/127.14.116.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.116.252:33275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.116.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:52:51.575511 5149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:52:51.576583 5149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:52:51.578397 5068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } attempt: 2
I20260619 06:52:51.579181 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.578903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48716)
0619 06:52:51.579041 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.579108 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 06:52:51.579948 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.579730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48718)
0619 06:52:51.579836 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.579885 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 06:52:51.580112 5068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:33275: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
I20260619 06:52:51.584069 5143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } attempt: 2
I20260619 06:52:51.584825 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.584563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48732)
0619 06:52:51.584668 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.584719 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 06:52:51.585067 5157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:52:51.585361 5156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:51.585296 5149 server_base.cc:1034] running on GCE node
W20260619 06:52:51.585072 5159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:52:51.585681 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.585448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48748)
0619 06:52:51.585555 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.585608 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 06:52:51.585810 5143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.116.252:33275: Network error: Client connection negotiation failed: client connection to 127.14.116.252:33275: connect: Connection refused (error 111)
I20260619 06:52:51.586227 5149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:52:51.587335 5149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:52:51.588541 5149 hybrid_clock.cc:648] HybridClock initialized: now 1781851971588512 us; error 31 us; skew 500 ppm
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:52:51.591341 5149 init.cc:377] Logged in from keytab as kudu/127.14.116.252@KRBTEST.COM (short username kudu)
I20260619 06:52:51.592271 5149 webserver.cc:460] Webserver started at http://127.14.116.252:42695/ using document root <none> and password file <none>
I20260619 06:52:51.592830 5149 fs_manager.cc:359] Metadata directory not provided
I20260619 06:52:51.592927 5149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:52:51.593158 5149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:52:51.595219 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/data/instance:
uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5"
format_stamp: "Formatted at 2026-06-19 06:52:51 on dist-test-slave-9x3b"
server_key: "ccd4d7a45e329a4c620124ac241cab8b"
server_key_iv: "fe9624738c3e6472762f775ab051fd94"
server_key_version: "encryptionkey@0"
I20260619 06:52:51.595831 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/wal/instance:
uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5"
format_stamp: "Formatted at 2026-06-19 06:52:51 on dist-test-slave-9x3b"
server_key: "ccd4d7a45e329a4c620124ac241cab8b"
server_key_iv: "fe9624738c3e6472762f775ab051fd94"
server_key_version: "encryptionkey@0"
I20260619 06:52:51.599503 5149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:52:51.602030 5166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:52:51.603183 5149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:52:51.603317 5149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/data,/tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/wal
uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5"
format_stamp: "Formatted at 2026-06-19 06:52:51 on dist-test-slave-9x3b"
server_key: "ccd4d7a45e329a4c620124ac241cab8b"
server_key_iv: "fe9624738c3e6472762f775ab051fd94"
server_key_version: "encryptionkey@0"
I20260619 06:52:51.603432 5149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:52:51.633070 5149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:52:51.635943 5149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:52:51.636171 5149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:52:51.643059 5149 rpc_server.cc:307] RPC server started. Bound to: 127.14.116.252:33275
I20260619 06:52:51.643337 5218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.116.252:33275 every 8 connection(s)
I20260619 06:52:51.644042 5149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/data/info.pb
I20260619 06:52:51.646659 5219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:52:51.647037 14803 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2eB_D1/build/debug/bin/kudu as pid 5149
I20260619 06:52:51.647181 14803 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2eB_D1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781851695409368-14803-0/minicluster-data/master-2/wal/instance
I20260619 06:52:51.647480 14803 external_mini_cluster.cc:1428] Setting key e6fefd8e7418b066482b0e860e3681a1
I20260619 06:52:51.651716 5219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
I20260619 06:52:51.661991 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.648829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.14.116.252:33275)
0619 06:52:51.649259 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.649288 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.649332 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:52:51.650086 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.650119 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.650564 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.650926 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.651660 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.653084 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.654241 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.654825 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.654944 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.657704 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.657710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.657719 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.657744 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.659327 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.659806 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.659816 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.659820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.659922 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.660197 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.660202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.660204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.660563 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.660755 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.661026 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.661281 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":114,"threads_started":1}
I20260619 06:52:51.662968 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.653638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51686 (local address 127.14.116.254:42807)
0619 06:52:51.653765 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.653771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.654045 (+ 274us) server_negotiation.cc:407] Connection header received
0619 06:52:51.654229 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.654233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.654311 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.654396 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.655077 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.655795 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.656766 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.656984 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.657071 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.659533 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.659537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.659541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.659571 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.661040 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.661645 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.661650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.661652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.661736 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.662011 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.662016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.662018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.662252 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.662376 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.662709 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.662819 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:52:51.662981 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.653573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.254:42807 (local address 127.0.0.1:51686)
0619 06:52:51.653930 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.653967 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:51.654095 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:51.654452 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:51.654457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:51.654768 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:51.654935 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.654943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.655964 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.655967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.656637 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.656648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.657143 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.657148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.657270 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.657818 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:51.657830 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:51.659396 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:51.661310 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.661320 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.661325 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.661543 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.661838 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.661841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.661843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.661924 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.662387 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:51.662394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:51.662501 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 06:52:51.662634 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 06:52:51.662821 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260619 06:52:51.664542 5219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.254@KRBTEST.COM
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
I20260619 06:52:51.673173 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.664347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.14.116.254:42807)
0619 06:52:51.664455 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.664460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.664471 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:52:51.664563 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.664567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.664604 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.664671 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.665535 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.666182 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.667116 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.667314 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.667389 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.669684 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.669688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.669690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.669734 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.671436 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.671953 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.671956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.671957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.672026 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.672306 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.672311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.672312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.672523 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.672748 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.672913 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.673001 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:52:51.673364 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.665139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.14.116.253:46353)
0619 06:52:51.665257 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.665260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.665273 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:52:51.665364 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.665366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.665402 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.665468 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.665861 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.666514 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.667454 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.667768 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.667880 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.670129 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.670139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.670142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.670164 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.671697 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.672161 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.672163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.672164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.672230 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.672506 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.672510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.672512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.672718 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.672810 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.673111 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.673226 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:52:51.673483 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.665032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.253:46353 (local address 127.0.0.1:41106)
0619 06:52:51.665202 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.665214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:51.665279 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:51.665510 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:51.665516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:51.665656 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:51.665740 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.665745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.666615 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.666618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.667322 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.667330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.667889 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.667894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.667999 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.668395 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:51.668408 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:51.669953 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:51.671800 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.671806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.671810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.672049 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.672336 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.672339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.672341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.672419 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.672834 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:51.672839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:51.672932 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 06:52:51.673048 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 06:52:51.673176 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 06:52:51.674286 5219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:52:51.676967 5143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } attempt: 3
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.252@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
I20260619 06:52:51.679683 5068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } attempt: 3
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.116.253@KRBTEST.COM
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.253@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
I20260619 06:52:51.684158 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.674740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48758)
0619 06:52:51.674930 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.674946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:51.675003 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:51.675273 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:51.675278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:51.675423 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:51.675489 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.675494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.676311 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.676315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.676900 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.676908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.677584 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.677587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.677661 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.678098 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:51.678114 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:51.680153 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:51.682060 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.682064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.682067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.682290 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.682550 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.682554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.682557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.682638 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.683049 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:51.683054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:51.683133 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 06:52:51.683230 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:52:51.683362 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 06:52:51.684336 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.674818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.14.116.253:46353)
0619 06:52:51.674929 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.674935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.674949 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:52:51.675103 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.675105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.675140 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.675196 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.675900 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.676429 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.677185 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.677722 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.678311 (+ 589us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.680390 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.680396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.680398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.680430 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.682167 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.682640 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.682644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.682646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.682724 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.683019 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.683022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.683025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.683218 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.683416 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.684074 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.684168 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 06:52:51.684708 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.674814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.14.116.252:33275)
0619 06:52:51.675044 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.675049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.675063 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:52:51.675099 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.675102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.675138 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.675238 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.675600 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.676215 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.677031 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.677368 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.677477 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.680291 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.680298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.680303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.680329 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.681941 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.682385 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.682388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.682389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.682459 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.682726 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.682729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.682730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.682927 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.683018 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.683877 (+ 859us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.684293 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
Jun 19 06:52:51 dist-test-slave-9x3b krb5kdc[4987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781851971, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.116.254@KRBTEST.COM for kudu/127.14.116.252@KRBTEST.COM
I20260619 06:52:51.687930 5219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5: Bootstrap starting.
I20260619 06:52:51.688043 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.677321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48762)
0619 06:52:51.677409 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.677424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:51.677494 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:51.678018 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:51.678022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:51.678215 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:51.678314 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.678321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.679494 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.679498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.680263 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.680275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.681732 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.681736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.681823 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.682247 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:51.682260 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:51.683670 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:51.686395 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.686401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.686403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.686650 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.686941 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.686944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.686946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.687035 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.687491 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:51.687495 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:51.687591 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 06:52:51.687753 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:52:51.687894 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 06:52:51.688104 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.677397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.14.116.252:33275)
0619 06:52:51.677765 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.677776 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.677790 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:52:51.677838 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.677843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.677885 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.678426 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.678500 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.679323 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.680402 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.681604 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.683775 (+ 2171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.683869 (+ 94us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.683872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.683875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.683891 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.686272 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.686753 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.686756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.686758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.686839 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.687130 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.687136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.687141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.687381 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.687489 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.687822 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.687937 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260619 06:52:51.690372 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.680050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.116.252:33275 (local address 127.0.0.1:48766)
0619 06:52:51.680259 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 06:52:51.680272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:52:51.680349 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:52:51.681042 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:52:51.681046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:52:51.681239 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:52:51.681351 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.681362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.682509 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.682513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.683409 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:52:51.683420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.684137 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.684146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:52:51.684250 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.685040 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:52:51.685056 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:52:51.686551 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:52:51.688963 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.688965 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.688967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.689152 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.689450 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:52:51.689453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:52:51.689459 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:52:51.689563 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:52:51.689974 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:52:51.689976 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:52:51.690066 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 06:52:51.690161 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:52:51.690264 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260619 06:52:51.690510 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.680472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.14.116.252:33275)
0619 06:52:51.680784 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.680794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.680807 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:52:51.680854 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.680859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.680902 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.681468 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.681585 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.682378 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.683628 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.683999 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.686649 (+ 2650us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.686764 (+ 115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.686766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.686769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.686788 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.688851 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.689253 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.689257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.689260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.689345 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.689659 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.689662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.689664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.689845 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.689950 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.690257 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.690362 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260619 06:52:51.691761 5219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:51.692646 5219 log.cc:824] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:51.693012 14803 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:52:51.693496 5068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae: Bootstrap starting.
I20260619 06:52:51.693553 5143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0: Bootstrap starting.
I20260619 06:52:51.694198 5042 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "05159fc0c25e44168cf3b6b90f8e3aae"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51698
I20260619 06:52:51.695806 5068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:51.696065 5143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0: Neither blocks nor log segments found. Creating new log.
I20260619 06:52:51.696417 5068 log.cc:824] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:51.696825 5143 log.cc:824] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:52:51.697556 5219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5: No bootstrap required, opened a new log
I20260619 06:52:51.698400 5068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae: No bootstrap required, opened a new log
I20260619 06:52:51.698931 5143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0: No bootstrap required, opened a new log
I20260619 06:52:51.701349 5068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } }
I20260619 06:52:51.701575 5068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05159fc0c25e44168cf3b6b90f8e3aae, State: Initialized, Role: FOLLOWER
I20260619 06:52:51.701905 5068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } }
I20260619 06:52:51.702283 5219 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } }
I20260619 06:52:51.702528 5219 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bfb2294e8a4eb4a90003370dbc4bb5, State: Initialized, Role: FOLLOWER
I20260619 06:52:51.702410 5229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.702778 5229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:51.702828 5219 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } }
I20260619 06:52:51.702785 5143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } }
I20260619 06:52:51.702977 5143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0, State: Initialized, Role: FOLLOWER
I20260619 06:52:51.703290 5068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:51.703264 5143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } }
I20260619 06:52:51.703362 5042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:52:51.703449 5042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:51.704092 5219 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:51.704475 5143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:52:51.704798 5042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } }
I20260619 06:52:51.705796 5042 leader_election.cc:288] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [CANDIDATE]: Term 1 election: Requested vote from peers 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 (127.14.116.253:46353), c2bfb2294e8a4eb4a90003370dbc4bb5 (127.14.116.252:33275)
I20260619 06:52:51.706383 5117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0"
I20260619 06:52:51.706600 5117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:51.708000 5117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05159fc0c25e44168cf3b6b90f8e3aae in term 1.
I20260619 06:52:51.708432 5230 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.708690 5230 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:51.708850 5019 leader_election.cc:302] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05159fc0c25e44168cf3b6b90f8e3aae, 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0; no voters:
I20260619 06:52:51.709137 5229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:52:51.709373 5229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [term 1 LEADER]: Becoming Leader. State: Replica: 05159fc0c25e44168cf3b6b90f8e3aae, State: Running, Role: LEADER
I20260619 06:52:51.709685 5229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } }
I20260619 06:52:51.711005 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5"
I20260619 06:52:51.711233 5193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:52:51.711349 5229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05159fc0c25e44168cf3b6b90f8e3aae. Latest consensus state: current_term: 1 leader_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.711440 5229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [sys.catalog]: This master's current role is: LEADER
W20260619 06:52:51.711776 5021 tablet.cc:2369] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:52:51.712363 5245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:51.712453 5245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:51.709488 5000 master_runner.cc:186] Error getting master registration for 127.14.116.253:46353: OK, instance_id { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" instance_seqno: 1781851971420596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:52:51.714498 5000 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:52:51.712692 5193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05159fc0c25e44168cf3b6b90f8e3aae in term 1.
W20260619 06:52:51.716190 5275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:51.716281 5275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:51.716472 5231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.716581 5231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:51.723979 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.708241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.14.116.254:42807)
0619 06:52:51.708355 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.708361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.710963 (+ 2602us) server_negotiation.cc:407] Connection header received
0619 06:52:51.712968 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.712971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.713061 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.713214 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.714284 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.715038 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.716457 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.716830 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.716909 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.719692 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.719696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.719703 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.719729 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.721428 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.722021 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.722025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.722027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.722120 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.722416 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.722420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.722421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.722626 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.722743 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.723598 (+ 855us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.723783 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:52:51.727890 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.708247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.14.116.253:46353)
0619 06:52:51.708363 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.708369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.712322 (+ 3953us) server_negotiation.cc:407] Connection header received
0619 06:52:51.717827 (+ 5505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.717830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.717875 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.717926 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.718806 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.719559 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.722228 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.722568 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.722641 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.724353 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.724359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.724363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.724389 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.726036 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.726457 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.726461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.726463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.726546 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.726832 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.726835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.726837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.727074 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.727188 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.727539 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.727688 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:52:51.728461 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:51.709185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.14.116.252:33275)
0619 06:52:51.710712 (+ 1527us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.710716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.713131 (+ 2415us) server_negotiation.cc:407] Connection header received
0619 06:52:51.713343 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.713347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.713395 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.713448 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.714376 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.715094 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.722005 (+ 6911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:51.722371 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:51.722452 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:51.724165 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:52:51.724169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:52:51.724171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:52:51.724195 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:52:51.725808 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.726958 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.726961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.726963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.727046 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:52:51.727356 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:52:51.727359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:52:51.727361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:52:51.727582 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:52:51.727745 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:51.728194 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 06:52:51.728301 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1459}
I20260619 06:52:51.730525 5274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:52:51.731273 5274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 06:52:51.733635 5271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:52:51.733709 5271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:52:51.740221 5193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [term 1 FOLLOWER]: Refusing update from remote peer 05159fc0c25e44168cf3b6b90f8e3aae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:51.740996 5117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [term 1 FOLLOWER]: Refusing update from remote peer 05159fc0c25e44168cf3b6b90f8e3aae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:52:51.741860 5229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:51.742300 5229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:52:51.746246 5231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05159fc0c25e44168cf3b6b90f8e3aae. Latest consensus state: current_term: 1 leader_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.746381 5231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:51.751049 5230 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05159fc0c25e44168cf3b6b90f8e3aae. Latest consensus state: current_term: 1 leader_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.751192 5230 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:51.760658 5229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.760828 5229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:51.762051 5230 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.762176 5230 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:51.764230 5278 mvcc.cc:205] Tried to move back new op lower bound from 7298465676239986688 to 7298465676126064640. Current Snapshot: MvccSnapshot[applied={T|T < 7298465676239986688}]
I20260619 06:52:51.765863 5231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.765983 5231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:52:51.766505 5229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05159fc0c25e44168cf3b6b90f8e3aae" member_type: VOTER last_known_addr { host: "127.14.116.254" port: 42807 } } peers { permanent_uuid: "5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0" member_type: VOTER last_known_addr { host: "127.14.116.253" port: 46353 } } peers { permanent_uuid: "c2bfb2294e8a4eb4a90003370dbc4bb5" member_type: VOTER last_known_addr { host: "127.14.116.252" port: 33275 } } }
I20260619 06:52:51.766984 5229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae [sys.catalog]: This master's current role is: LEADER
I20260619 06:52:51.767633 5279 mvcc.cc:205] Tried to move back new op lower bound from 7298465676239986688 to 7298465676126064640. Current Snapshot: MvccSnapshot[applied={T|T < 7298465676239986688}]
I20260619 06:52:51.767824 5281 mvcc.cc:205] Tried to move back new op lower bound from 7298465676239986688 to 7298465676126064640. Current Snapshot: MvccSnapshot[applied={T|T < 7298465676239986688}]
I20260619 06:52:51.769239 5274 catalog_manager.cc:1347] Generated new cluster ID: 440bd9b3f3a44b9292f1c0e9f53c0555
I20260619 06:52:51.769304 5274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:52:51.819839 5274 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:52:51.821153 5274 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:52:51.840355 5274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05159fc0c25e44168cf3b6b90f8e3aae: Generated new TSK 0
I20260619 06:52:51.841260 5274 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:52:51.878928 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.875709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.14.116.253:46353)
0619 06:52:51.875855 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.875860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.877818 (+ 1958us) server_negotiation.cc:407] Connection header received
0619 06:52:51.878026 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.878030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.878082 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.878136 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.878790 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41116: BlockingRecv error: recv got EOF from 127.0.0.1:41116 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 06:52:51.880672 5072 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.877199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.14.116.254:42807)
0619 06:52:51.877314 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.877319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.879686 (+ 2367us) server_negotiation.cc:407] Connection header received
0619 06:52:51.879850 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.879853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.879901 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.879951 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.880498 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51716: BlockingRecv error: recv got EOF from 127.0.0.1:51716 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 06:52:51.881388 5225 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.876556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.14.116.252:33275)
0619 06:52:51.876698 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.876703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.879418 (+ 2715us) server_negotiation.cc:407] Connection header received
0619 06:52:51.879521 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.879525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.879577 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.879709 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.881281 (+ 1572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48784: BlockingRecv error: recv got EOF from 127.0.0.1:48784 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 06:52:51.890095 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.888516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.14.116.253:46353)
0619 06:52:51.888630 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.888635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.888949 (+ 314us) server_negotiation.cc:407] Connection header received
0619 06:52:51.889180 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.889184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.889258 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.889314 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.889967 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41118: BlockingRecv error: recv got EOF from 127.0.0.1:41118 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 06:52:51.896260 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.894536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.14.116.253:46353)
0619 06:52:51.894652 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.894656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.895208 (+ 552us) server_negotiation.cc:407] Connection header received
0619 06:52:51.895404 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.895408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.895458 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.895510 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.896153 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41124: BlockingRecv error: recv got EOF from 127.0.0.1:41124 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 06:52:51.902148 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.900442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.14.116.253:46353)
0619 06:52:51.900556 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.900561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.901174 (+ 613us) server_negotiation.cc:407] Connection header received
0619 06:52:51.901380 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.901383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.901438 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.901494 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.902042 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41140: BlockingRecv error: recv got EOF from 127.0.0.1:41140 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 06:52:51.918285 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.916038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.14.116.253:46353)
0619 06:52:51.916185 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.916190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.916456 (+ 266us) server_negotiation.cc:407] Connection header received
0619 06:52:51.916664 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.916668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.916725 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.916798 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.918172 (+ 1374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41146: BlockingRecv error: recv got EOF from 127.0.0.1:41146 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 06:52:51.930505 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.928588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.14.116.253:46353)
0619 06:52:51.928717 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.928723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.929379 (+ 656us) server_negotiation.cc:407] Connection header received
0619 06:52:51.929577 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.929581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.929636 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.929689 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.930380 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41154: BlockingRecv error: recv got EOF from 127.0.0.1:41154 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 06:52:51.949908 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.948805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.14.116.253:46353)
0619 06:52:51.949006 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.949010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.949081 (+ 71us) server_negotiation.cc:407] Connection header received
0619 06:52:51.949196 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.949199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.949246 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.949320 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.949749 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41160: BlockingRecv error: recv got EOF from 127.0.0.1:41160 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260619 06:52:51.984943 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:51.983896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.14.116.253:46353)
0619 06:52:51.984057 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:51.984061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:51.984198 (+ 137us) server_negotiation.cc:407] Connection header received
0619 06:52:51.984322 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:51.984325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:51.984374 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:51.984457 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:51.984842 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41176: BlockingRecv error: recv got EOF from 127.0.0.1:41176 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 06:52:52.052299 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:52.051074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.14.116.253:46353)
0619 06:52:52.051229 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:52.051234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:52.051410 (+ 176us) server_negotiation.cc:407] Connection header received
0619 06:52:52.051518 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:52.051520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:52.051568 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:52.051695 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:52.052177 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41184: BlockingRecv error: recv got EOF from 127.0.0.1:41184 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 06:52:52.183542 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:52.182433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.14.116.253:46353)
0619 06:52:52.182579 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:52.182584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:52.182744 (+ 160us) server_negotiation.cc:407] Connection header received
0619 06:52:52.182849 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:52.182852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:52.182899 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:52.182985 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:52.183398 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41192: BlockingRecv error: recv got EOF from 127.0.0.1:41192 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 06:52:52.442626 5150 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:52:52.441558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.14.116.253:46353)
0619 06:52:52.441708 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:52.441712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:52.441820 (+ 108us) server_negotiation.cc:407] Connection header received
0619 06:52:52.441919 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:52.441921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:52.441967 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:52.442037 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:52:52.442486 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41202: BlockingRecv error: recv got EOF from 127.0.0.1:41202 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:52:52.717857 5275 catalog_manager.cc:1259] Loaded cluster ID: 440bd9b3f3a44b9292f1c0e9f53c0555
I20260619 06:52:52.717991 5275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0: loading cluster ID for follower catalog manager: success
I20260619 06:52:52.719625 5275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0: acquiring CA information for follower catalog manager: success
I20260619 06:52:52.720676 5275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e0cc1e6bdcd4a0f886c9d2bfe2dcbe0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:52:52.734794 5271 catalog_manager.cc:1259] Loaded cluster ID: 440bd9b3f3a44b9292f1c0e9f53c0555
I20260619 06:52:52.734862 5271 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5: loading cluster ID for follower catalog manager: success
I20260619 06:52:52.736430 5271 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5: acquiring CA information for follower catalog manager: success
I20260619 06:52:52.737521 5271 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2bfb2294e8a4eb4a90003370dbc4bb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:52:52.960727 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:52.956663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.14.116.253:46353)
0619 06:52:52.956901 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:52.956908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:52.957009 (+ 101us) server_negotiation.cc:407] Connection header received
0619 06:52:52.957181 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:52.957184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:52.957326 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:52.957444 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:52.957708 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.958250 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.959132 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.959470 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.959568 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:52.959920 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:52.960342 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:52.960436 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:52.960491 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 06:52:52.960534 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
I20260619 06:52:52.967015 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:52.963556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.14.116.252:33275)
0619 06:52:52.963821 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:52.963825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:52.963838 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:52:52.963984 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:52.963987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:52.964095 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:52.964187 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:52.964356 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.964883 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.965620 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.965826 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.965896 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:52.966122 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:52.966623 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:52.966710 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:52.966789 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 06:52:52.966833 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260619 06:52:52.973798 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:52.970144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.14.116.252:33275)
0619 06:52:52.970245 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:52.970250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:52.970318 (+ 68us) server_negotiation.cc:407] Connection header received
0619 06:52:52.970520 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:52.970523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:52.970564 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:52.970638 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:52.970887 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.971634 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.972623 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.972944 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.973035 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:52.973338 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:52.973414 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:52.973500 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:52.973583 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 06:52:52.973641 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:52:52.973979 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:52.969808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.14.116.253:46353)
0619 06:52:52.969928 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:52.969933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:52.970463 (+ 530us) server_negotiation.cc:407] Connection header received
0619 06:52:52.970588 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:52.970598 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:52.970668 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:52.970736 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:52.971093 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.971555 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.972774 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.973119 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.973214 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:52.973496 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:52.973572 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:52.973633 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:52.973795 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 06:52:52.973863 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:52:52.974615 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:52:52.969814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.14.116.254:42807)
0619 06:52:52.970057 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 06:52:52.970063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:52:52.970260 (+ 197us) server_negotiation.cc:407] Connection header received
0619 06:52:52.970479 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:52:52.970485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:52:52.970644 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:52:52.970746 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:52:52.971165 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.971926 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.972980 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:52:52.973266 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:52:52.973370 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:52:52.973880 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:52:52.974196 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:52:52.974326 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:52:52.974395 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 06:52:52.974448 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260619 06:52:52.977542 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 5000
I20260619 06:52:52.983724 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 5075
I20260619 06:52:52.989046 14803 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2eB_D1/build/debug/bin/kudu with pid 5149
2026-06-19T06:52:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3116 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6158 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277573 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST